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ABSTRACT 


In the last 20 years, the Department of Defense has seen a general reduction in 
the acquisition workforce yet experienced unprecedented growth in spending for 
services. The government has increased the level of scrutiny on service 
contracts highlighting the need to consistently follow policy when documenting 
contractor performance in the Contractor Performance Assessment Reporting 
System (CPARS). 

This report used statistical analysis to examine 715 Army service 
contractor performance reports in CPARS in order to answer the following 
questions: (1) Are government contracting professionals submitting contractor 
performance narratives in accordance with CPARS guidelines? (2) What is the 
added value of the contractor performance narratives beyond the value of the 
objective scores for performance? (3) What is the statistical relationship between 
the narratives and the objective scores? (4) Do the interview findings contradict, 
support, or enhance the findings for the three questions above? (5) What 
conclusions or recommendations can we draw from the answers to the questions 
above? 

The findings revealed inconsistencies between the objective scores and 
narratives entered into CPARS. This report discusses the findings and concludes 
with eight recommendations to improve documenting contractor performance. 
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I. INTRODUCTION 


A. BACKGROUND 

The purpose of this chapter is to introduce the system designated by the 
Department of Defense (DOD) for reporting contractor past performance, called 
Contractor Performance Assessment Reporting System (CPARS). The DOD 
obligated approximately $360 billion in fiscal year 2012 for contracts for supplies 
and services (Government Accountability Office [GAO], 2013). In order to 
support best practices for government acquisition, the Office of Federal 
Procurement Policy (OFPP) directs in Federal Acquisition Regulation (FAR) Part 
42 that federal contracting agencies shall use CPARS as their primary tool for 
documenting contractor past performance. 

This MBA project focuses on Army services contracts, specifically, the 
narrative section of contractor performance assessment reports, and advances 
the research conducted by Flart, Stover, and Wilhite (2013), which was focused 
on the CPARS contractor report cards. This research project generated a local 
database of 715 Army service contracts that were used to correlate the success 
of contracts based on objective scores of six characteristics. 

This is the eighth research project in a series of MBA projects relating to 
services acquisitions. Apte, Dixon, and Rendon are collaborating on research 
that focuses on the value of CPARS narratives, the correlation of narratives to 
objective scores, and the narratives’ effect upon the source selection process. 

Government contracting professionals use contractor past performance 
information (PPI) during source selection and contract administration. The FAR 
requires that contracting officers consider contractor past performance as one 
evaluation factor for awarding contracts. Contractor PPI can be the factor that 
determines whether a particular contractor will be awarded government business. 
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B. PROBLEM STATEMENT 


CPARS is a system that was designed to be used by contracting 
professionals for submitting and retrieving contractor past performance 
information. There have been many discrepancies and failures specified in 
recent Government Accountability Office (GAO) reports with the utilization of 
CPARS. Government contracting agencies are failing to properly document 
contractor PPI within the schedule requirements mandated by the OFPP. In a 
2009 report, data analyzed from 2007 showed that DOD contracting components 
completed required contractor past performance report cards less than half of the 
time (GAO, 2009). Subsequently, the DOD increased its focus on training and 
education for contracting professionals, which led to an increase in contractor 
performance assessments being completed and submitted to the Past 
Performance Information Retrieval System (PPIRS). In the last published report 
during the research period of this project (GAO, 2013), the GAO noted significant 
gains in completion rates. Fifty-six percent of required reports were completed in 
2011, while 74% were completed in 2013. However, according to the same 
report, over half of these reports were submitted late. In addition, many CPARS 
report cards contain narratives that are either insufficiently detailed or conflict 
with their associated objective scores. Late reports lacking sufficient accurate 
information provide less-than-optimal information to the contracting professionals 
who rely on these report cards for source selection and contract administration 
purposes. 

C. PURPOSE 

Government contracting professionals submit to CPARS information on 
contractors related to their contract performance. This information is in the form of 
objective scores in five categories: quality, schedule, cost control, business 
relations, and management of key personnel. In addition to these five categories, 
there is a specific contractor evaluation section in which the government evaluator 
writes a descriptive narrative of the contractor’s performance. 
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The purpose of this research is to attempt to determine the value of 
contractor performance assessment report narratives for services contracts by 
comparing the relationships between narratives and objective scores in order to 
recommend improvements to the CPARS contractor performance information 
documentation process. These recommendations regarding the process aim to 
improve the CPARS report card product, which should lead to greater and more 
effective utilization of the CPARS system for source selection and contract 
administration purposes. 

D. RESEARCH QUESTIONS 

The primary question addressed by this research project is as follows: 
Does the CPARS report card written narrative section provide value to the 
contractor performance evaluation process? This research focuses on five 
research questions that explore the nature of CPARS narratives: 

1. To what degree are government contracting professionals 
submitting contractor performance narratives to CPARS in 
accordance with the guidelines provided in the CPARS user’s 
manual? 

2. What is the added value of the contractor performance narratives 
beyond the value of the objective scores for performance? 

3. What is the statistical relationship between the sentiment contained 
in the narratives and the objective scores for contractor 
evaluations? 

4. To what degree do the interview findings contradict, support, or 
enhance the findings for the three research questions above? 

5. What conclusions or recommendations can we draw from the 
answers to the research questions above? 
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E. METHODOLOGY 

This research examines the value of CPARS report card narratives for 
service contracts as they relate to their associated objective scores. The 
methodology includes a literature review, data analysis (including both sentiment 
and statistical analysis), and interviews with government agency contracting 
professionals. 

The literature review was conducted for the purpose of examining and 
considering knowledge available from the existing body of literature on the 
subjects of federal contracting and government contractor performance. The 
GAO and OFPP have released several studies and memoranda on the subject of 
contractor past performance as it relates to the government contracting process. 
This research was based in large part upon these documents and previous 
research performed by Hart, Stover, and Wilhite (2013) that explored the drivers 
of successful service contracts. Building on their research, this project 
conducted a review of literature about the service contract management process 
and the Contractor Performance Assessment Reporting System. 

The first phase of data analysis performed was a sentiment analysis of 
715 Army service contract CPARS report card narratives. The CPAR Quality 
Checklist was used as a basis for developing the criteria for the categories and 
values for the analysis {CPAR Quality Checklist, n.d.). Narratives were assigned 
several scores relating to their quality, robustness, compliance with directions in 
the CPAR Quality Checklist, and their value and content compared to their 
related objective scores from the CPARS report cards. 

The second phase of data analysis included a statistical analysis of the 
relationship between the scores provided by the sentiment analysis and their 
associated objective scores. This analysis looked at correlating relationships 
between the various sentiment scores and the various objective scores for the 
same CPARS report to find meaningful relationships between them and to reveal 
the extent of the value of the narratives. 
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In support of the data analysis, interviews with contracting professionals 
from two DOD contracting agencies were conducted. These interviews consisted 
of a series of questions asking the subjects about how they use and to what 
extent they value CPARS and other sources of contractor past performance 
information and to what extent they value the narratives compared to the 
performance objective scores. 

F. BENEFITS AND LIMITATIONS 

The purpose of this research is to attempt to determine the value of 
contractor performance assessment report narratives for services contracts by 
comparing the relationships between narratives and objective scores. Potential 
benefits to this research include a better understanding of the value of the 
CPARS narratives and the extent to which DOD contracting professionals value 
them relative to other sources of contractor past performance information. This 
study will provide information based on data indicating how closely written 
narratives correlate to their associated objective scores. It will also show how 
well DOD approving officials and the individuals contributing to the narratives 
comply with the directions for completing a CPARS narrative as instructed in the 
CPAR Quality Checklist {CPAR Quality Checklist, n.d.). 

There are clearly identifiable limitations to the research being conducted. 
First, this research project is limited to the 715 Army service contracts analyzed 
in a previous Naval Postgraduate School (NPS) MBA project (Hart et al., 2013). 
The selection criteria for those 715 contracts was limited by Army service 
contracts, “contracts only submitted by Mission Installation Contracting 
Commands (MICCs), and contracts from only five MICC offices” (from Hart et al., 
2013, p. 5). 
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The next limitation is the dollar threshold for reporting contractor 
performance information in CPARS. DOD CPARS policy provides different dollar 
thresholds for mandatory reporting of contractor past performance (PPI) 
information. Service contracts must report contractor PPI starting at dollar 
amounts greater than $1,000,000 (Department of the Navy [DON], 1997). This 
means contractor PPI for services contracts may not be reported if dollar 
amounts are below $1 million. This limits the research database to either service 
contracts above $1 million or service contracts below the threshold that DOD 
contracting professionals chose to enter into CPARS. 

Another limitation of this research is the fact that DOD contracting 
professionals may not enter PPI into CPARS even though it is required. As of 
the completion of this project, the most recent report (GAO, 2013) shows this 
lack of performance with 56% and 74% of CPARS reports completed in 2011 and 
2013, respectively. These limitations are discussed further in later chapters. 

G. SCOPE AND ORGANIZATION OF RESEARCH 

This research report consists of five chapters organized in the following 
manner. Chapter I contains background information, a problem statement, the 
purpose of the research project, research questions, the methodology of 
research, the benefits and limitations of the project, and the scope and 
organization of the research project. Chapter II is a literature review that provides 
an overview of the DOD acquisition process for service contracts, the contractor 
performance assessment reporting system, the increase in scrutiny of federal 
contracting agencies, and previous studies that led us to this research. Chapter 
III contains the methodology for how the research was performed, organized, and 
analyzed. This chapter also discusses the purpose and scope of interview 
questions posed to DOD contracting professionals. Chapter IV provides an 
analysis of the data and presents the answers to the research questions. 
Chapter V summarizes the research project, provides conclusions, and 
addresses potential areas for additional research. 
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H. SUMMARY 


This chapter provided the background information leading up to the 
research project, a problem statement, the purpose of the research project, 
research questions, the research methodology used to analyze the data, benefits 
and limitations of the research, and the scope and organization of the overall 
project. Chapter II provides the literature review of the DOD acquisition services 
contracting process, the CPARS evaluation system, and previous studies 
conducted by NPS in relation to services contracts. 
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II. LITERATURE REVIEW 


A. INTRODUCTION 

The purpose of this chapter is to provide an introduction to the DOD 
acquisition process for services, with a focus on Army services contracts. This 
review attempts to familiarize the reader with the necessary information to 
understand the contracting structure, contracting terms, roles, and 
responsibilities of the acquisition team, and the system that the DOD has 
implemented for documenting and retrieving contractor past performance 
information, called the Contractor Performance Assessment Reporting System 
(CPARS). This chapter provides a discussion about the purpose of CPARS as 
well as current issues associated with CPARS. The chapter includes a look at 
reports submitted by a number of government authorities that point out the 
successes and failings of various government contracting agencies in complying 
with laws and directives specifying how and to what extent they are to utilize 
CPARS for documenting contractor past performance information. 

B. THE DOD ACQUISITION PROCESS FOR SERVICE CONTRACTS 

To understand the DOD acquisition process, the reader first needs to 
understand what the term acquisition means. The Federal Acquisition 
Regulation (FAR) defines the term acquisition as follows: 

acquiring by contract with appropriated funds of supplies or 
services (including construction) by and for the use of the Federal 
Government through purchase or lease, whether the supplies or 
services are already in existence or must be created, developed, 
demonstrated, and evaluated. Acquisition begins at the point when 
agency needs are established and includes the description of 
requirements to satisfy agency needs, solicitation and selection of 
sources, award of contracts, contract financing, contract 
performance, contract administration, and those technical and 
management functions directly related to the process of fulfilling 
agency needs by contract. (FAR, 2014, Part 2.101b) 
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The basic definition is that the government has a need and fulfills that need by 
purchasing a product or service through a defined process using a set of rules. 

All DOD acquisitions—services and non-services alike—begin the 
contracting process with the same fundamental step: identify a bona fide need or 
requirement. The customer, usually an organization or an entity within an 
organization, identifies a product or service that is essential for that organization 
to complete its mission or function. The customer must be specific when 
describing the agency need for the contracting officials to accurately identify the 
requirement and begin the contracting process. According to FAR Part 11, 
Describing Agency Needs, the customer must, to the maximum extent possible, 
adhere to the following: 

(i) State requirements with respect to an acquisition of supplies or 

services in terms of - 

(A) Functions to be performed; 

(B) Performance required; or 

(C) Essential physical characteristics, (from FAR, 2014, Part 

11 . 002 ) 

An example of a bona fide need is as follows. The Army Mission and 
Installation Contracting Command (AMICC) received a request from a customer 
to purchase janitorial services. The customer has funding available in the current 
fiscal year and is requesting 12 months of services for one building with a 
requested contract start date two months from the submission of the 
requirement. The services consist of the following tasks: sweep floors daily, mop 
floors with soap and water one time per week, wax floors one time per quarter, 
remove trash from garbage bins every Thursday, clean and sanitize bathrooms 
daily, and other specified tasks. The service requirement is clearly defined with 
funding available to obligate expenses and provides the contracting official with 
sufficient information to begin the acquisition process, and so a bona fide need is 
apparent. 
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The following request for maintenance services would be rejected due to 
its not meeting the bona fide need requirement. The AMICC receives a request 
for M1A1 Abrams tank maintenance services from Fort Drum’s 10th Mountain 
Division. This request would be denied as the 10th Mountain does not have 
tanks on its Table of Organization and Equipment (TO&E). A unit cannot request 
maintenance services for equipment it does not have. Alternatively, a request by 
the U.S. Army Armor School at Fort Knox might have a bona fide need for tank 
maintenance services because they train Army and Marine Corps personnel on 
the Abrams tank. 

Once a bona fide need has been identified, the service acquisition process 
can begin. The service acquisition plan is a comprehensive strategy that 
identifies the relationship between the plan’s phases and the work to be 
performed, lists the milestones to be completed, and provides the overall 
approach that management will follow to mitigate risk while meeting the service 
requirements (Neuman, 2013b). 

The service acquisition process can be identified by its three phases: 
Plan, Develop, and Execute. These phases break down into seven distinct 
processes that build upon the previous process. As Beers (2011) wrote, “The 
Planning phase, as depicted in Steps 1, 2, and 3, lays the foundation for the 
services acquisition” (from Beers, 2011, p. 56). According to Beers (2011), 
“During the development phase—Steps 4 and 5—a requirements roadmap 
process is used to define performance objectives and standards, allowable 
variations, and method of performance assessment. In the execution phase— 
Steps 6 and 7—the team puts all the customer’s planning and development 
efforts into action” (from Beers, 2011, p. 57). Figure 1 explains the seven stages 
of the service acquisition process. 
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• Select Ri|ht Contnctor 

• Award Contract 

• Roll Out Strategy 


Execute 

Figure 1. The Services Acquisition Plan (from Beers, 2011, p. 56) 


Phase 1: Plan 

The first phase of the services acquisition process is the planning phase. 
This phase is extremely important as it is the foundation for the service 
acquisition. The planning phase contains three steps: (1) form the team, 
(2) review current strategy, and (3) market research. 

Step 1: Form the Team 

During Step 1 of this phase, the agency head is responsible for 
designating who will be a member of the acquisition team, provide them with the 
guidance on how to proceed, and ensuring that the plan complies with FAR 
requirements (FAR, 2014, Part 7.103[i]). The acquisition team consists of anyone 
who will be responsible for significant portions of the service acquisition 
throughout its life cycle (FAR, 2014, Part 7.102[b]). No single person has the 
requisite knowledge, technical proficiency, time, or resources to perform all the 
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functions necessary to plan, source, award and administer a service contract, but 
an acquisition team made up of experts in functional areas can. Acquisition team 
members can include, but are not limited to, the program manager, contracting 
officer, contracting officer’s representative (COR), and others with expert 
functional area knowledge like finance, legal, engineering, small business 
administration, quality assurance, logisticians and the customer/end user (DOD, 
2012 ). 

Program Manager: The program manager (PM) is the single person 
tasked with the responsibility to manage the acquisition team while balancing the 
acquisition cost, schedule, and performance for that program (Neuman, 2013a). 
PMs perform essential functions such as attending program reviews, defending 
the program or fighting for additional funding, fighting requirements creep, and 
coordinating with senior DOD and congressional staffers to ensure compliance 
with statutory requirements and congressional intent. Given the importance of 
the PM position, it is unfortunate that previous research by Apte and Rendon 
(2007) found that “in many service acquisitions, a project manager or project 
team is not assigned and the contracting officer assumes the responsibilities of 
the project manager or project team leader” (from Hart et al., 2013, p. 10). 

Contracting Officer: The contracting officer (KO), as defined by Neuman 
(2013b), is: 

responsible for providing contracting support required for meeting 
the project’s cost, schedule, and performance objectives while in 
compliance with the statutory requirements and agency regulations 
ensuring performance of all necessary actions for effective 
contracting, ensuring compliance with the terms of the contract, and 
safeguarding the interests of the United States Government in its 
contractual relationships. (Neuman, 2013b, p. 34). 

Basically, the KO works to support the PM to meet the program’s goals, but the 
KO must adhere to all laws, regulations, and agency policies. 

Contracting officers have specific responsibilities defined by regulations. 
According to FAR 1.602-2, a contracting officer shall; 
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(a) Ensure that the requirements of 1.602-1 (b) have been met, 
and that sufficient funds are available for obligation; 

(b) Ensure that contractors receive impartial, fair, and equitable 
treatment; 

(c) Request and consider the advice of specialists in audit, law, 
engineering, information security, transportation, and other 
fields as appropriate; 

(d) Designate and authorize, in writing and in accordance with 
agency procedures, a contracting officer’s representative 
(COR) on all contracts and orders other than those that are 
firm-fixed price, and for firm-fixed-price contracts and orders 
as appropriate, unless the contracting officer retains and 
executes the COR duties, (from FAR, 2014, Part 1.602-2) 

Army contracting officials can either be civilian employees authorized to 
serve in 1102 positions (Office of Personnel Management, 1983), or military 
personnel serving in Functional Area 97A, Contracting and Industrial 
Management Officer (Jones, 2007), or Military Occupational Specialty 51C, 
Acquisition, Logistics, and Technology Contracting Non-Commissioned Officer 
(U.S. Army, 2011). 

In the past decade. Army acquisition officials have increasingly operated 
in joint environments. This has required Army contracting personnel to interact 
with contracting specialists from the other services and other governmental 
agencies. While all DOD personnel operate under the same FAR and Defense 
Federal Acquisition Regulation Supplement (DFA'RS), each DOD organization 
has its own supplement to the FAR, and each organization outside the DOD may 
operate under different guidance promulgated by its parent organization (e.g.. 
Army FAR Supplement [AFARS], Air Force FAR Supplement [AFFARS], Navy 
Marine Corps Acquisition Regulation Supplement [NMCARS], Department of 
State Acquisition Regulation [DOSAR], and Department of Veterans Affairs 
Acquisition Regulation [VAAR]; FAR, 2014, Part 1.301). 

Contracting Officer Representative: The contracting officer 
representative (COR) is not a contracting officer, nor does he/she hold any 
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contracting authority. The FAR specifically states that a COR “has no authority 
to make any commitments or changes that affect price, quality, quantity, delivery, 
or other terms and conditions of the contract nor in any way direct the contractor 
or its subcontractors to operate in conflict with the contract terms and conditions” 
(from FAR, 2014, Part 1.602-2[d][5]). The function of a COR is to be the eyes 
and ears of the KO. Ideally, the COR would be a technical expert and assist with 
monitoring contractor performance on a specific contract. In practice, it is more 
likely that a COR will not have a high degree of technical expertise in the field of 
monitoring contractor performance. 

Functional Area Expert: Functional area experts have a specific set of 
skills and provide the acquisition team with a wide range of talent that they can 
tailor to each service contract. They provide an essential service by reducing the 
workload and technical knowledge required by the contracting officer. Functional 
area experts include, but are not limited to, financial managers who keep the 
project on budget, price/cost estimators who analyze the price/cost of a project, 
legal officers, engineers who have technical knowledge, auditors, small 
business/competition advocates, quality assurance representatives, information 
technology advisors, and logisticians (Beers, 2011). Service contracts can range 
from a simple janitorial contract to highly complex multi-year helicopter 
maintenance and repair services that are performed on multiple bases in the 
continental United States; overseas bases in Europe, Africa, and Korea; and 
warzones like Afghanistan and Iraq. 

Step 2: Review Current Strategy 

Step 2 begins with the acquisition team creating a baseline and then 
analyzing the current service strategy (Beers, 2011). This consists of looking at 
historical data, identifying problems, projecting any type of modification to the 
mission, and getting the customer/end user to define the key performance 
outcomes that they expect from this contract. This is by far the most important 
part of Step 2. The customer must be able to define what they expect this 
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contract to accomplish as there can be differences between what is said and 
what is understood. 

For example, a customer requested a janitorial cleaning services contract 
to cover portable toilets at a specific location. The contracting officer 
understands that this contract is to cover cleaning the portable toilets and 
removing waste, but goes back to the customer to define key requirements to 
make sure everyone has the same view of the performance requirements. The 
customer states that they want the vendor to check the level of toilet paper in 
each portable toilet and resupply each unit as necessary. The customer also 
requires that the services contract cover the removing of waste water and trash 
from the hand washing station next to the portable toilets and the resupplying of 
the hand sanitizer solution and paper towels at that station. What the customer 
originally stated as the requirement and what he/she thought he/she was getting 
was different from what the contracting officer understood from the original 
request. This is why it is important for the customer and acquisition team to 
define the requirements and expected performance outcomes. 

Step 3: Market Research 

As Beers (2011) wrote, “The team analyzes the marketplace to assess 
current technology and business practices, competition and small business 
opportunities, and existing and potential new sources of providing the service; 
the team then determines if commercial buying practices can be used” (from 
Beers, 2011, p. 56). Ideally, the customer or end-user would have conducted 
market research prior to giving the requirements to the contracting command. 
Market research can provide the customer and the contracting officer with ideas 
about the types of services required, possible vendors to provide those services, 
estimated pricing and cost data, and help defining what the customer really 
needs. One of the main reasons for doing market research is to determine 
whether a service is commercially available or is a government-unique service. 
Commercial services tend to be more defined and have more vendors to 
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compete for government business. Government-unique services can be more 
complicated and require more effort to define the requirement. 

Phase 2: Develop 

The second phase of the services acquisition process is the Development 
phase. This phase consists of Step 4, Requirements Definition, and Step 5, 
Acquisition Strategy, in which the acquisition team develops the overall plan to 
procure the services. The requirements and performance outcomes that the 
customer provided are examined and refined further. Then the team determines 
the appropriate path to contract for the required services. 

Step 4: Requirements Definition 

This step allows the acquisition team to define the performance objectives, 
determine an acceptable variation, and choose the method of measuring 
performance (Beers, 2011). This process leads the team to develop the 
Performance Work Statement (PWS). The PWS should tie directly to the 
requirements and is used to define how the vendor will be evaluated with regard 
to CPARS objective scores. 

The following is an example of defining requirements. The Army Mission 
and Installation Contracting Command (AM ICC) received a request from a 
customer to purchase aircraft maintenance services. This is poorly defined 
because there are many different types of aircraft that require different types of 
maintenance. This could mean ordering spare parts, engine repair, repair of 
proprietary hardware, repair of electrical or electronic systems, or some other 
type of maintenance. The requirement does not say where the maintenance 
services will be performed. When does the customer need the services? How 
many aircraft will be serviced or how much maintenance service does the 
customer require? The requirement provided to the contracting official is very 
vague, which does not allow him/her to determine what the customer wants or 
needs. The contracting official will need to discuss the requirements with the 
customer to define the requirements. 
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step 5: Acquisition Strategy 

This step is designed to select the contract type that is the most beneficial 
to the government to mitigate risk, provide for the proper amount of incentive to 
the vendor to complete the contract objectives, and select between lowest price 
and tradeoffs where the best value may be a higher priced solution (Beers, 
2011). The requirements will tell the vendor what the performance metrics are, 
not the specific methods for how the vendor will complete the job. The customer 
and acquisition team care only about the performance outcomes, not how the 
vendor accomplishes the task. This allows the vendor to be innovative about 
how they design a solution. 

Phase 3: Execute 

The third phase of the services acquisition plan is the Execution phase. 
This phase consists of Step 6, Execute Strategy, and Step 7, Performance 
Management, which pulls together everything that the customer and acquisition 
team have developed up to this point and culminates in the requirement going 
out to industry in the form of a request for proposals (RFP). The final part of 
Phase 3 is concerned with contract performance and administration processes 
designed to close out or terminate a contract. 

Step 6: Execute Strategy 

This step involves the creation of synopsis and solicitation documents that 
“formally communicates to industry the customer’s requirements and business 
plan” (from Beers, 2011, p. 57). The acquisition team may release a draft RFP to 
solicit industry feedback on the feasibility of the requirements and to get a feel for 
industry interest in performing this contract. Once the requirement is ready for 
release to the public, vendors will review the documents and create a solution to 
meet those objectives. The acquisition team may establish a competitive range 
to limit the number of applicants to an economical, efficient, and manageable 
level as their time and resources are limited and cannot process 100, 50, or even 
10 proposals if the requirements are complex. 
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The acquisition team will evaluate each proposal against the requirements 
specified in the RFP and the standards and performance measures included in 
the Performance Work Statement (PWS). The team will not evaluate proposals 
against each other. The acquisition team will recommend to the source selection 
authority (SSA) the proposal that best meets the RFP requirements. The final 
decision on selecting the best proposal rests with the SSA. The SSA may be 
one or more levels above the contracting officer. The contracting officer will then 
inform all the non-selects of the results. If there is some discrepancy with the 
contract award, the non-selects will have 10 days from the contract award to 
protest the award. At this point, the acquisition team will move into the 
administrative or management step. 

Step 7: Performance Management 

As Beers (2011) observed, this step “involves two key areas: 
administering contract requirements, such as invoicing and payments, and 
managing the relationships and expectations of both the contractor and 
customers in meeting the terms of the contract and achieving the required 
mission performance results” (from Beers, 2011, p. 57). Many in the contracting 
world argue that this step is where the real work begins. 

Contractor performance on the contract must be monitored, which is 

where the COR aids the contracting officer. The COR will communicate with the 

contractor and contracting officer regularly to provide progress reports and help 

solve problems as they arise. Contracts can be terminated for default (T4D), 

terminated for convenience (T4C) of the government, or closed out upon 

successful completion. While contracts should be closed out within six months, 

there are situations in which contracts remain open for decades while the 

government and contractor disagree over payments, terms and conditions, and 

settlement of any court appeals. An example is Boeing determining what the 

final labor rate, overhead rate, and time and material rates are for a cost 

reimbursement type contract. Boeing must wait until all other product lines have 

provided data so the company can break down those costs to the relevant 
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contract and develop these rates for reimbursement. The government will then 
review those costs for accuracy which can take up to a year or more on complex 
contracts. If there is a discrepancy, it can go to the contract agency head, the 
GAO, or the Armed Services Board of Contract Appeals, which can add years to 
the case. The contract cannot be closed out until every last discrepancy is 
resolved, even if it’s as simple as a $0.01 difference between the contracted 
price and the contractor invoice. 

Contractor performance must be monitored regularly to comply with 
contract administration requirements. Contract administration can cover short 
periods of time or govern multiple periods, depending on the type of contract. A 
contract with a short period of performance may see the source selection, 
contract award, contract performance, and closed out periods all occur within the 
same fiscal year. In this instance, a contracting officer would provide the CPARS 
with an evaluation of that contractor’s performance on that particular contract 
based upon the evaluation criteria set forth in the contract. A contract that 
crosses fiscal years, has multiple option years of the same contract, or is a multi¬ 
year contract (e.g., ship building, military construction), may require a CPARS 
evaluation report on the contractor’s performance on an annual or more regularly 
scheduled basis and a final CPARS evaluation when the contract closes out. 
Military personnel move on a regular basis so the PM, KO, or COR who starts on 
a contract source selection may be different from the PM, KO, or COR who 
administers the contract, who might differ from the PM, KO, or COR who closes 
out the contract. Diligent monitoring of contractor performance is the only way to 
accurately evaluate and rate a contractor in CPARS over the life of the contract. 

Accurate CPARS report cards, objective scores, and descriptive narratives 
are important because they are used during the government’s source selection 
process. As described in Figure 1, the Services Acquisition Plan, the 
government will execute the acquisition strategy by selecting a contractor and 
monitoring performance. Contractor past performance can play a significant role 
in determining eligibility and competitiveness in the source selection process. 
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Contractor past performance can demonstrate, but is not limited to, contractor 
responsiveness to customer needs, reliability, contractor’s past performance on 
relevant contracts, government/contractor business relationships, disqualifying 
conditions (debarment, suspension, termination for default, failure to meet 
standards), contractor/subcontractor business relationships, and factors to 
mitigate potential risk. Given the role that past performance information plays in 
the source selection process, it is important to review the CPARS system. The 
next section provides an overview of CPARS, the narrative and issues 
surrounding CPARS information. 

C. CONTRACTOR PERFORMANCE ASSESSMENT REPORTING 
SYSTEM 

The CPARS is used by all federal agencies to evaluate contractor 
performance. The purpose behind this system is to give federal procurement 
personnel a way to assess “a contractor’s performance, both positive and 
negative, and provide a record on a given contract during a specified period of 
time” (from Hart et al., 2013, p. 20). CPARS report cards are segmented into five 
distinct business sectors: Systems, Services, Operations Support, Fuels, and 
Information Technology (DON, 1997). The key business sector “Services” can 
be subdivided into Professional/Technical & Management Support Services, 
Repair & Overhaul, and Installation Services. The excerpt below is from the 
CPARS manual and describes the types of contracts that are considered 
“Service” contracts. 

Services: Generally, all contracted services except those related to 
“Science & Technology,” “Construction & Architect—Engineering 
Services,” and “Health Care.” 

Professional/Technical & Management Support Services: 

Includes all consultant services—those related to scientific and 
technical matters (e.g., engineering, computer software engineering 
and development), as well as those related to organizational 
structure, human relations, etc. Includes office administrative 
support services (e.g., any basic or applied research that will result 
in new or original works, concepts, or applications, but does not 
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include contract advice on the feasibility of such research, as well 
as evaluation of research results. 

Repair & Overhaul: Services related to the physical repair and 
overhaul of aircraft ground vehicles, etc., and any associated 
subsystems or components. Includes condition evaluations of 
individual items received for repair or overhaul, but does not include 
evaluations of the feasibility or the benefits of the overall project. 

Does not include ship repair and overhaul which is included under 
the Systems sub-sector on Shipbuilding. 

Installation Services: Includes services for grounds maintenance 
(grass cutting, shrubbery maintenance or replacement, etc.). 
Includes services related to cleaning, painting, and making minor 
repairs to buildings and utilities services, etc. Includes contracted 
security and guard services. Includes installation and maintenance 
of fencing. Includes minor electrical repairs (e.g., replacing outlets, 
changing light bulbs, etc.). Includes minor road surface repairs 
(patching cracks, filling in potholes, etc.). Includes relocation of 
individual telephone lines and connections. Includes snow 
removal. (But, see also “Construction & Architect/Engineering 
Services” and Information Technology” for the services covered by 
those business areas.), (from DON, 1997, p. A1-2) 

The Services Business Sector is a very diverse collection of services. 
Contracting professionals may not be familiar or experienced with such a wide 
array of contracted services so it is important for CORs, KOs, PMs, and the 
contract management team to work together to ensure that the contractor’s 
performance is clearly and accurately recorded in the CPARS report card. 

Past performance information entered into CPARS must be accurate and 
timely as it is primarily used by source selection officials to award or not award a 
contract based on a contractor’s strengths and weaknesses during Step 6, the 
Execute Strategy phase of the Services Acquisition Plan. As explained in the 
CPARS manual, “The value of CPARS to a future source selection team is 
inextricably linked to the care the program manager takes in preparing a quality 
narrative to accompany the CPAR scores” (from DON, 1997, p. 1). 

The CPARS evaluation assesses at least five distinct areas of contractor 
performance supported by objective data presented in Block 20 (DON, 1997). 
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These areas are Quality of Product/Service, Schedule, Cost Control, Business 
Relations, and Management of Key Personnel. Objective scores are used by the 
acquisition team to rate contractor performance in these five areas. There are 
five possible scores for each of these five areas. The five possible scores are 
unsatisfactory, marginal, satisfactory, very good, and exceptional (also shown in 
Table 3). 

CPARS includes a variance assessment that assesses the variance 
between current costs and schedules and government estimates. Scoring for 
Block 18 and 19 must be in accordance with Appendix B, SPECIFIC EXAMPLES 
OF NARRATIVE STATEMENTS TO AVOID FROM THE CPAR QUALITY 
CHECKLIST (N.D.) and Evaluation Ratings Definitions (DOD, 2011, p A2-1), and 
is described in detail in Chapter III—Research Methodology. 

Finally, CPARS includes a program manager narrative that is a statement 
of the facts regarding the contractor’s performance on a particular contract. The 
program manager is responsible for generation and content of the narrative as 
the PM must sign and date this form prior to sending it to the KO for review 
(DON, 1997). The narrative comments in Block 20 should directly trace back to 
the corresponding objective scores in Blocks 18 and 19 of the CPARS report 
card (DON, 1997). 

Consistent and objective evaluations of contractor performance are 
essential, which is why a series of checks and balances were designed within the 
CPARS process (Hart et al., 2013). As stated in the CPASRS manual, “Each 
assessment must be based on objective facts and be supportable by program 
and contract management data, such as cost performance reports, customer 
comments, quality reviews, technical interchange meetings, financial solvency 
assessments, and functional performance evaluations” (from DON, 1997, p. 2). 

Before the performance assessment can be finalized, it must be forwarded 
to the contractor for review and comment. This ensures the past performance 
information report card is finalized only after the contractor has the opportunity to 
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provide feedback to the assessing official. The contractor is not required to 
respond but must have the opportunity to provide inputs to the CPARS 
evaluation. Only after this has happened can the approving official finalize the 
report. The approving official is someone in a position at least one level above 
the program manager to help ensure impartiality and objectivity (DON, 1997). 
Even with these checks and balances in place, the Army and the DOD as a 
whole have had below-standards records for CPARS compliance. Table 1 
shows the compliance rates for the Army and the larger DOD with a 49% and 
56% in 2011 but a much improved compliance rate of 71% and 74%, 
respectively, for 2013 (GAO, 2013, p. 9). 


DOD Compliance Rate as of 


Comoonent 

2011-0ct-03 

2012-SeD-28 

2013-ADr-01 

Air Force 

82% 

82% 

80% 

Navy 

66% 

69% 

72% 

Army 

49% 

60% 

73% 

Other DOD 

32% 

61% 

71% 

Total DOD 

56% 

66% 

74% 


Table 1. Percentage of Required Assessments Submitted to the 
Past Performance Information Retrieval System (PRIRS) 
(from GAO, 2013, p. 9) 


The following section addresses several issues identified by the GAO and 
the Office of Federal Procurement Policy (OFPP) in the area of documenting 
contractor past performance information in CPARS and their steps to correct and 
improve the process. 

D. INCREASED SCRUTINY OF FEDERAL CONTRACTING AGENCIES 

An increased focus on contractor past performance began in 1994 with 
the Federal Acquisition Streamlining Act (FASA), in which Congress directed that 
government contracting agencies should use contractor past performance 
information as one of the award decision evaluation factors. As a result of the 
FASA, the FAR was updated to reflect this new requirement, and the OFPP 
began releasing guides and memoranda for the purpose of instructing federal 
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contracting professionals in the best practices for collecting, submitting, and 
utilizing contractor past performance information. 

One of the biggest obstacles in the way of achieving more reliably 
widespread and frequent use of contractor past performance information in the 
contracting process was the lack of a centralized reporting system. In 2002, the 
DOD directed that the PPIRS, a web-enabled contractor past performance 
information database and retrieval tool, was its single authorized retrieval system 
for past performance information. However, a GAO report from 2009 revealed 
that “PPIRS data for fiscal years 2006 and 2007 indicates that only a small 
percentage of contracts had a documented performance assessment” (from 
GAO, 2009, Executive Summary). Table 2 shows the disparity between the total 
numbers of contracts requiring assessments versus contracts with an actual 
completed assessment. In 2007, the Air Force led the way with 47%. In 2009, 
the FAR began requiring all federal agencies to submit all of their contractor 
performance evaluations to PPIRS. 


Estimated Contracts Requiring an Assessment That Had an Assessment In Fiscal Year 2007 


Estimated contracts 

Contracts with an 


Department/agency 

requiring an assessment 

assessment 

Percent 

Air Force 

2,795 

1,300 

47 

Navy 

3,879 

1,622 

42 

Army 

6,145 

1,971 

32 

Other DOD 

1,408 

303 

22 

Homeland Security 

4,131 

535 

13 

NASA 

3,706 

1,093 

29 

Energy 

840 

183 

22 

Total percentage 

22.904 

7.007 

31 


Source: GAO analysis of data from DOD. FPDS-NG. and PPIRS. 


Table 2. DOD CPARS Requirement versus Actual Reporting 
(from GAO, 2009, p. 12) 


In 2007, the GAO released a report. Use of Contractor Performance 
Information, detailing its study of how contractor past performance information 
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may be considered in the contracting process and what issues government 
contracting agencies have encountered with the use of this information. The 
report identifies many of the complexities that go into the process of evaluating 
contractor past performance and how these complexities have affected GAO 
protest decisions (GAO, 2007). 

Contractor past performance information is fundamentally important to the 
contracting process from the beginning during the pre-award period through the 
end of contract performance. The GAO report looked at several ways in which 
government contracting finds use in considering contractor past performance. 
These include source selection, responsibility determinations, surveillance of 
performance under the current contract, and suspension and debarment (GAO, 
2007). 

A GAO report from 2009 revealed the results of a review of 62 
government contract solicitations from fiscal years 2007 and 2008 for which 
contractors’ past performance information was considered (GAO, 2009). The 
researchers also interviewed 121 contracting officials, asking them about how 
their agencies utilize contractor past performance information. The GAO study 
showed that although contractor past performance information was being 
reviewed, to include the use of the PPIRS system, that “factors other than past 
performance, such as technical approach or cost, were the primary factors for 
contract award decisions” (from GAO, 2009, Executive Summary). While the 
FAR does allow contracting officials broad discretion for how heavily to weight 
contractor past performance relative to other evaluation factors in their source 
selection evaluation, the GAO study revealed that “a majority of officials told us 
their reluctance to rely more on past performance was due, in part, to their 
skepticism about the reliability of the information and difficulty assessing 
relevance to specific acquisitions” (from GAO, 2009, p. 8). 

The OFPP followed the GAO report in 2009 with a memorandum for the 

chief acquisition officers and senior procurement executives that acknowledged 

the “fragmented methods” that agencies used to collect and maintain contractor 
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performance information. The OFPP stated that agencies “maintain evaluations 
in internal data systems that are not available to acquisition officers outside that 
agency” (from OFPP, 2009, p. 1). This was occurring despite the FAR 
requirement that contractor performance information be shared between 
agencies. The OFPP issued new requirements to agencies pursuant to FAR 
subpart 42.15 changes effective July 2009 mandating submission of contractor 
performance records to PPIRS. The OFPP encouraged government contracting 
agencies to designate specific individuals to the task of ensuring that “accurate, 
complete, and timely information is submitted to PPIRS” (from OFPP, 2009, p. 2). 

In 2011, the OFPP issued a memorandum to share the findings of its 
review of agency compliance with its 2009 directives and to provide additional 
recommendations for continued improvement in the collection of contractor past 
performance information. Researchers looked at nearly 700 performance reports 
submitted by the 10 federal contracting agencies that make up the bulk of the 
federal contracting, together accounting for 94% of federal contract obligations 
for fiscal year 2009 (OFPP, 2011). This review took place during the period in 
which contracting agencies were migrating to the Contractor CPARS for the 
purpose of submitting their contractor performance information to PPIRS. 

The OFPP found that in 2009, the DOD had conducted required past 
performance evaluations on about 50% of contract awards and that the quality of 
those submitted were of varying quality with most deemed lacking in sufficient 
information. They evaluated the contractor performance report narratives based 
on how well they addressed the four required rating factors: quality of the product 
or service, ability to control cost, ability to meet schedule, and quality of business 
relations (OFPP, 2011). The DOD has used CPARS for submitting contractor 
past performance reports since 2004 (GAO, 2013). 

The OFPP review of performance reports submitted by the DOD revealed 
the following: 53.1% contained sufficient narrative for quality of product/service, 
51% contained sufficient narrative for schedule control, 21.9% contained 
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sufficient narrative for cost control, and 50% contained sufficient narrative for 
business relations (OFPP, 2011). 

The OFPP issued guidance to improve the quality of contractor past 
performance reporting. Agencies were directed to establish roles and 
responsibilities for those individuals appointed to review and submit performance 
reports and to facilitate proper training for all acquisitions personnel. A new 
increased focus on training and accountability was expected to significantly 
improve contractor performance reporting. Agencies are required to establish an 
agency point of contact accountable for disseminating guidance, facilitating 
training, developing oversight, and identifying improvements to the submission 
process. They were directed to institute a performance report review process for 
the purpose of monitoring their quality (OFPP, 2011). 

In 2013, the OFPP took additional steps to improve the quality of 
contractor performance assessments. In March, it released a memorandum 
establishing a baseline for compliance with contractor performance reporting 
standards and setting agency performance targets for fiscal years 2013 through 
2015 (OFPP, 2013). 

A 2013 GAO report shows that measures taken by the OFPP and the FAR 
Council have led to higher rates of submission for performance assessments 
within the DOD. In 2007, fewer than half were completed. Submission of 
required assessments increased from 56% in October 2011 to 74% by April 
2013. Flowever, the report shows that the DOD is still failing to complete 
assessments on time. As Figure 2 shows, a large portion of the required 
assessments are completed late (i.e., after the 120-day requirement; GAO, 
2013). 
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Figure 2. 
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DOD CPARS Reporting Compliance 2010-2012 (from 
GAO, 2013, p. 10) 


The GAO report focuses on quantifying the effectiveness of the measures 
taken by the DOD to improve the quality and timeliness of its contractor 
performance reports that it submitted via CPARS. DOD requirements for 
contractor performance reports (as specified in the CPARS Guide) specifies that 
performance assessments must be completed and finalized within 120 days from 
the end of the contractor evaluation period and that the assessing official is 
responsible for preparing and finalizing the report. The report indicates that 
“although the CPARS Guide does not currently specify standards for 
completeness ... a recent proposed change to the FAR will address 
completeness by providing minimum government-wide standards for past 
performance rating elements. Specifically, the proposed rule requires that all 
assessments address, at a minimum, quality of product or service, timeliness, 
and management or business relations” (from GAO, 2013, p. 8). As of June 
2013, there were no formally established standards for quality and completeness 
of CPARS performance narratives. 

CPARS data is not and should not be the only source of information used 
to determine contractor past performance (DOD, 2012). Other sources of PPI 
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are references provided by said contractor, references provided by other federal 
contracting personnel, and other information not in CPARS. One of the 
limitations of CPARS is that federal contracting personnel are not required to 
enter PPI into CPARS if the dollar amount of the specific contract does not meet 
the minimum reporting threshold, as described in Figure 3. While the 
government may elect to enter PPI into CPARS below these thresholds, this is 
not common practice and, as a result, leads to gaps in PPI data below the 
$1 million threshold for service contracts. 


Business Sector 

Dollar Threshold 

Systems 

> $5,000,000 

Ship Repair & Overhaul 

> $500,000 

Services 

> $1,000,000 

Health Care 

> $150,000 

Operations Support 

> $5,000,000 

Fuels 

> $150,000 

Information Technology 

> $1,000,000 


Figure 3. DOD Reporting Thresholds 


E. PREVIOUS STUDIES 

This is the seventh study of a series of research projects aimed at 
services contract management. The original research began in 2006 when Apte, 
Ferrer, Lewis, and Rendon examined a growing trend of DOD acquisition 
workload over the previous decade. As noted by Flart, Stover, and Wilhite 
(2013), one of the major observations of this research was a 66% increase in 
services contracting since 1999 but no corresponding increase in contract 
management personnel (Apte et al., 2006). 

The second research project was an exploration of supply chain 

management of service contracts (Apte & Rendon, 2007). In general terms, the 

research focused upon how the Air Force used the Air Education and Training 
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Center (AETC) and Acquisition Management and Integration Center (AMIC) 
models (Hart et al., 2013). Apte and Rendon observed that these processes 
enabled cradle-to-grave handling of a service acquisition but allowed for a 
communication failure as program managers are not on site (Hart et al., 2013). 

The third research project differed from the previous ones in that this was 
based on a survey of Air Force and Navy service contracts (Hart et al., 2013). 
Apte, Apte, and Rendon focused upon program management issues and 
approaches of service contracting (Apte et al., 2008). This was relevant to 
research conducted by Hart, Stover, and Wilhite but is not directly relevant to the 
narrative evaluation in CPARS. 

The fourth research project focused on services contract management 
was published in 2009 and expanded upon the third research project by including 
the Army in the survey. Again, problems were noted with the acquisition team 
and communication issues (Hart et al., 2013). The major takeaway from this 
research is that most acquisition professionals believed that “their organizations 
[did not have] sufficient positions and that the available positions were [not] 
adequately filled” (from Hart et al., 2013, p. 29). Like the previous research 
projects, this improves the knowledge base for service contracts but is not 
directly related to the narrative evaluation in CPARS. 

The fifth research project, conducted by Apte, Apte, and Rendon in 2010, 
collected data from the previous studies and analyzed them for DOD 
management techniques for services acquisition (Hart et al., 2013). They 
focused upon improving the communication of contracting personnel in the Army, 
Air Force, and Navy. Like the previous research projects, this improves the 
knowledge base for service contracts but is not directly related to the narrative 
evaluation in CPARS. 

The sixth research project was completed in 2012 by Apte, Apte, and 
Rendon. This project focused upon the drivers of management practices of 
successful service contracts in the Army (Hart et al., 2013). Hagan, Spede, and 
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Sutton surveyed 168 key personnel to determine the definition of a successful 
service contract in the Navy (Hart et al., 2013). The outcome of this research 
was a correlation of modifications, protests, and communications between the 
acquisition team members. Like the previous research projects, this improves 
the knowledge base for service contracts but is not directly related to the 
narrative evaluation in CPARS. 

The seventh and most germane research project was completed in 2013 
by Hart, Stover, and Wilhite. The group explored the definition of a successful 
service contract by looking at the CPARS report cards stored within the PPIRS. 
A local database was created from 715 Army service contract CPARS report 
cards for the purpose of evaluating success determined by six CPARS evaluation 
areas: Quality of the Product/Service, Schedule, Cost Control, Business 
Relations, Management of Key Personnel, and Utilization of Small Business 
(Hart et al., 2013). 

Contractor evaluation scores for the six evaluation areas were assigned a 
numerical value by the researchers according to the adjectival rating scale in 
Table 3. The research calculated the average scores to find any correlation to 
success or failure for the six evaluation areas. This rating system and the 
associated database is the foundation for the research that is to be examined in 
Chapter III and Chapter IV of this research project. 


Rating 

Score 

Exceptional 

5 

Very Good 

4 

Satisfactory 

3 

Marginal 

2 

Unsatisfactory 

1 


Table 3. Area of Contract Evaluation Scores 
(from Hart et al., 2013, p. 44) 


Of the 715 Army service contracts reviewed, 22 were deemed failures 

because they received a rating of unsatisfactory or marginal for one or more of 
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the six evaluation areas previously listed (Hart et al., 2013). Table 4 shows the 
breakdown of the 715 contracts reviewed by Hart, Stover, and Wilhite (2013). 



Failures 

Success 

Total 

Failure Rate 

Contracts 

22 

693 

715 

3.08% 


Table 4. Total Contract Information (from Hart et al., 2013, p. 44) 


Hart, Stover, and Wilhite (2013) found that service contracts had an 
average failure rate of approximately 3% (see Table 4). They cited several 
factors that led to contract failure, but a common theme among them was a lack 
of proper training for source selection evaluators (SSEs). SSEs must be able to 
properly evaluate each contractor’s proposal according to the requirements in the 
RFP during the source selection process. Past performance information, to 
include objective ratings and descriptive narratives, plays an important role in the 
SSE’s evaluation of the contractor’s proposal. It is this past performance 
information that is the focus of our research project. 

The seven previous service contract-related research projects have 
provided the background and the insight to pursue this research project. CPARS 
report cards, PPI, and the database of 715 Army service contracts generated by 
Hart, Stover, and Wilhite form the core of the analysis of program manager 
narratives and their relation to the objective scores in CPARS. 

F. SUMMARY 

This chapter presented an overview of the DOD services acquisition 
process and the literature associated with said process. The chapter began with 
a presentation of the service contract management process as outlined in Beers’ 
2011 Services Acquisition Plan. Next, the chapter reviewed the CPARS to 
provide a framework for the database which is the basis for this research project. 
Finally, the chapter concluded with a review of previous research projects related 
to the service contracting field of study. 
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The next chapter discusses the research questions that this research 
attempts to answer, the methods used to conduct the research, the raw data to 
be analyzed, and the type of statistical analysis used to understand and explain 
the relationships of the data. 
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III. RESEARCH METHODOLOGY 


A. INTRODUCTION 

This chapter discusses how the research was conducted and how the 
authors answered the research questions. The chapter begins by describing 
how the objective scores and narratives are entered into CPARS and how it 
relates to a successful contract. Then the chapter contains an in-depth 
description of how that data is analyzed, including a description of the statistical 
methodology used to perform the analysis. The last part of this chapter describes 
the interviews that were conducted, along with the interview question 
development methodology. 

This research examines the value of CPARS report card narratives for 
service contracts as they relate to their associated objective scores. The 
methodology includes data analysis (including both sentiment and statistical 
analysis) and interviews with government agency contracting professionals. 

B. OBJECTIVE SCORES AND NARRATIVES 

The first phase of data analysis performed was a sentiment analysis of 
715 Army service contract CPARS report card narratives. The CPAR Quality 
Checklist was used as a basis for developing the criteria for the categories and 
values for the analysis {CPAR Quality Checklist, n.d.). Data from 715 CPARS 
report cards of service contracts from 2012 were made available as the basis of 
this continuing research (Hart, Stover, & Wilhite, 2013). Each sample provided a 
series of objective scores in a spreadsheet that was associated with each 
narrative. 

Table 5 provides an example of objective scores for the six evaluation 
areas used to rate a contractor’s performance. 
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Quality of 
Product or 
Service 

Schedule 

Cost 

Control 

Business 

Relations 

Management 
of Key 
Personnei 

Utilization 
of Smail 
Business 

Min 

Score 

Success 
or Faiiure 

VERY 

VERY 

VERY 

VERY 

VERY 

VERY 



GOOD 

GOOD 

GOOD 

GOOD 

GOOD 

GOOD 

4 

SUCCESS 

(4) 

(4) 

(4) 

(4) 

(4) 

(4) 




Tables. Example of Objective Scores 


Table 6 provides examples of narratives that describe in the appropriate 
amount of detail the performance of the contractor for that particular objective 
score. For example, an objective score of very good for the area of quality of 
product or service does not tell the full story. The narrative associated with an 
objective score of very good is meant to explain how the contractor maintained a 
very good level of quality via a quality control program that was operated 
efficiently and effectively throughout the life of the contract. 


Quality of Product/Service: This contractor has done a very good job overall. This particular 
task order is for the management Firm Fixed Priced piece in support of a major customer. 
Overall, quality of the products and services remains high and the contractor quality control 
program runs efficiently and effectively. 

Schedule: Contractor continues to provide products/services on time. Contractor continues to be 
very adaptive to unanticipated work requests. 

Cost Control: Contractor has historically produced accurate invoices. This is certified annually 
under the Contractor Manpower Reporting system. 

Business Relations: Contractor has very good Business Relations. The government has never 
encountered any difficulty in working with this contractor to resolve any issue. Contractor has 
three (3) subcontractors assisting in the performance of this operation. Contractor provided daily 
reports on all equipment under their control for maintenance. The contractor was effective in 
programs such as equal opportunity, employee incentive, energy conservation, safety, security, 
and upward mobility. 

Management of Key Personnel: All key personnel have performed very well. 

Utilization of Small Business: Contractor utilizes three small businesses as subcontractors on 
this task order. 

Overall Comments: Contractor continues to satisfy customers. The government frequently 
receives positive comments on the end product received from this contractor. Contractor provides 
a very good service through the cost plus contract. The government definitely would recommend 
awarding future contracts to this contractor. 

Given what I know today about the contractor's ability to execute what he promised in his 
proposal, I DEFINITELY WOULD award to him today given that I had a choice. 

Table 6. Example of Narratives Associated with Objective 

Scores 
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To analyze the narratives in the samples and see how they relate to the 
objective scores, a series of criteria had to be developed. The CPAR Quality 
Checklist was used as a basis for developing the criteria {CPAR Quality 
Checklist, n.d.). Narratives were assigned several scores relating to their quality, 
robustness, compliance with directions in the CPAR Quality Checklist, and their 
value and content compared to their related objective scores from the CPARS 
report cards. 

First, we answered a series of yes or no questions. These questions are 
listed as follows. 

First, does the narrative address all performance areas assessed as an 
objective score? This means that if one of the five objective scores was marked 
as not applicable, there would not need to be something written in the narrative 
specifically mentioning that area. Qtherwise, there should be something written 
in the narrative to address the score given for each area in the objective scores. 

Second, is the narrative based on objective data? In answering this 
question, objective data is defined as not being influenced by personal feelings or 
opinions of the person inputting the data, but only considering and representing 
facts. 

Third, is the narrative free of statements to avoid, as defined by the CPAR 
Quality Checklist? The following phrases (or those closely similar) qualify as 
narrative statements to avoid: “outside the scope of the contract,” “in our 
opinion,” “appeared,” “we believe,” “it is our hope,” “we are not happy,” “we did 
not like,” “we think,” “could be,” and “we hope.” Appendix B provides an 
expanded list of narrative statements to avoid. 

Fourth, is the narrative for each performance area of a robust and 
comprehensive nature, as described in the three examples of the CPAR Quality 
Checklist, labeled “Block 20 - insufficient narrative vs. a better way to write this”? 
Appendix C provides specific examples of insufficient narratives. 
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Fifth, could a contracting layman understand the work that has been 
performed? This will not require an opening statement giving a synopsis of the 
contract in the narrative because that would be part of a separate data field 
defined as a sample contract effort description within CPARS (Block 17) outside 
the narrative portion (Block 20). Rather, the question would answer whether 
there was an excessive use of jargon or acronyms in the narrative, to the point 
that even someone familiar with common terms associated with contracting 
would be unable to understand the discussion in the narrative. 

Once the yes or no questions have been scored, the following two 
questions would be answered with a possible score of 1 to 5. 

First, is the narrative beneficial to a user of the information above and 
beyond the objective scores? To earn a score of 1, the narrative provides an 
unsatisfactory amount of beneficial data to the user above and beyond what could 
be gleaned from looking over the objective scores assigned in Block 18. To earn a 
score of 2, the narrative provides a marginal amount of beneficial data to the user 
above and beyond what could be gleaned from looking over the objective scores 
assigned in Block 18. To earn a score of 3, the narrative provides a satisfactory 
amount of beneficial data to the user above and beyond what could be gleaned 
from looking over the objective scores assigned in Block 18. To earn a score of 4, 
the narrative provides a very good amount of beneficial data to the user above and 
beyond what could be gleaned from looking over the objective scores assigned in 
Block 18. To earn a score of 5, the narrative provides an exceptional amount of 
beneficial data to the user above and beyond what could be gleaned from looking 
over the objective scores assigned in Block 18. 

Second, does the narrative correlate to the objective scores that have 

been assigned? To earn a score of 1, more than one of the performance areas 

described in the narrative is contradictory to the objective scores assigned in 

Block 18. To earn a score of 2, one (but no more than one) of the performance 

areas described in the narrative is contradictory to the objective score assigned 

in Block 18. To earn a score of 3, the narrative is merely satisfactory in 
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describing accurately why the objective scores are assigned as they are in Block 
18. To earn a score of 4, the narrative is very good in describing accurately why 
the objective scores are assigned as they are in Block 18. To earn a score of 5, 
the narrative is exceptional in describing accurately why the objective scores are 
assigned as they are in Block 18. 

The next step in conducting our analysis of the narratives was to figure out 
the best way to go about handling the sheer volume of 715 report cards while at 
the same time thoroughly reviewing each report card. Software was an initial 
thought, but after looking at this option in depth, it became obvious that the 
process of learning how to use and manipulate the software to give the desired 
output would be more time consuming than it was worth. For this sample size, it 
turns out that it would be best to conduct an Inter-Rater Reliability Test (Gwet, 
2008) to determine a baseline agreement amongst the raters of the questions 
outlined previously, and then go through each narrative thoroughly by the raters. 
If this could prove that the raters were trained and could rate each sample 
similarly, than each of the 715 samples would not need to be analyzed by each 
researcher. Using this approach, the 715 samples could be split up amongst the 
raters and would yield similar results across each rater for the same sample. 

After conducting training on the procedures outlined above in conducting 
each score, a sample of 30 reports was randomly selected from the original 715 
and the reports were independently rated by each rater. Appendix A provides 
the results of the Inter-Rater Reliability Testing. 

To score each set of scores for the yes or no questions, a score of 3 was 
given if all three of the scores matched. The only other possibility for the yes or 
no questions is for two of the scores to match and the other to not match 
(highlighted in red). In this case, a score of 1 was assigned since two of the 
scores had a 50% rate of agreeing to the other scores while the remaining score 
had a 0% rate of agreeing to the other scores. In the end, the scores for each 
set of scores was summed and then divided by 90 to get the total agreeability 
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percentage in each category. The acceptable threshold was set at whether all 
raters could match scores for each sample with a minimum of 80% reliability. 

The results for the yes or no questions show the percentage of 
agreeability between each rater’s scores of the 30 randomly selected reports. 
There was 100% agreeability on whether the narrative addresses all 
performance areas assessed. There was 84.44% agreeability on whether the 
narrative was based on objective data. There was 97.877% agreeability on 
whether the narrative was free of statements to avoid. There was 80% 
agreeability on whether the narrative was robust and comprehensive. There was 
91.11% agreeability on whether a contracting layman could understand the work 
performed. 

For the questions that were assigned scores of 1 to 5, the scores show the 
percentage agreeability of whether the raters could agree in their results while 
allowing for a 1 point difference amongst raters with no penalty. The threshold 
was set at whether all raters could match scores for each sample within a 1 point 
difference at a minimum 80% reliability. If a set of scores all matched, a score 
of 3 was given for the set. If a set of scores had two occurrences of scores within 
1 (for example, 1, 2, 3 or 3, 4, 5), a score of 2 was given for the set. If a set 
of scores had only one occurrence of scores within 1 (example 2, 4, 5), a score 
of 1 was given for the set. If there had been an instance where scores of 
1, 3, and 5 had been given, a score of 0 would have been given for the set (this 
did not occur). 

The results show the percentage of agreeability between each of the 
rater’s scores within a score of 1 based on the 30 randomly selected samples. 
There was 93.33% agreeability within 1 on whether the narrative is beneficial 
above and beyond the objective scores. There was 80% agreeability within 1 on 
whether or not the narrative correlates to the objective scores assigned. 

Because all of the total agreeability percentages were able to meet the 
minimum 80% threshold, each rater went through ~230 of the remaining report 


40 



cards independently and the results were combined and analyzed as if all three 
raters had gone through all 715 report cards. The next section discusses the 
methodology for the statistical analysis of the database. 

C. METHODOLOGY FOR DATABASE STATISTICAL ANALYSIS 

An independent 2-sample test for Proportions was performed to find the 
statistical significance of the proportions of the outcomes for successful versus 
unsuccessful contracts for the questions that have only two possible outcomes 
(yes or no). The p value in the hypothesis test shows the level of statistical 
significance. A p value of less than .05 shows statistical significance. A p value of 
less than .01, therefore, shows a much higher degree of statistical significance 
than even the .05 values. 

A chi-square test was performed to show collectively whether the results 
for the questions that have five possible outcomes were statistically different for 
successful versus unsuccessful contracts. The p value used in the chi-square 
test shows the level of statistical significance and the results are interpreted in 
the same way as in the hypothesis test that compares two independent 
proportions (explained in the previous paragraph). The next section discusses 
the interview questions that were asked government contracting personnel. 

D. INTERVIEWS OF GOVERNMENT CONTRACTING PERSONNEL 

In support of the data analysis, interviews with contracting professionals 
from two DOD contracting agencies were conducted. These interviews consisted 
of a series of questions asking the subjects about how they use and to what 
extent they value CPARS and other sources of contractor past performance 
information and to what extent they value the narratives compared to the 
performance objective scores. 

Our research includes the study of human subjects and therefore, as 
directed by regulation from the Department of Health and Human Services, the 
research interviews required approval from an institutional review board (IRB) 
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under the guidance of the NPS Office for Human Research Protections (OHRP) 
specialist. The purpose of the IRB is to protect human subjects in research and 
to ensure that basic ethical principles are adhered to. 

We conducted interviews at two geographically diverse DOD contracting 
agencies. The interviewees had some combination of significant experience 
submitting contractor past performance information to CPARS and/or retrieving 
from the Past Performance Information Retrieval System (PPIRS) past 
performance information for use in contract administration or source selection. 
Interview subjects were chosen by their respective department heads. 

E. INTERVIEWS QUESTION DEVELOPMENT METHODOLOGY 

Interviews with contracting professionals who are experienced with the 
CPARS and PPIRS systems provide the requisite context and depth of 
knowledge needed to accurately and robustly analyze and interpret the statistical 
data derived from the CPARS narratives. The specific interview questions were 
designed to avoid, to the extent practicable, injection of bias by steering 
respondents. Interviewing subjects at two government agencies with significantly 
different missions and contract award types facilitates the collecting of 
information across a wide spectrum of knowledge of the CPARS/PPIRS systems. 
Interviewing individuals with varying roles in collecting, submitting, and utilizing 
contractor past performance information provides us with responses across a 
diverse spectrum of experience. A complete listing and explanation of the 
interview questions can be found in Appendix D. 

Following this second phase of research, the study moved into analyzing 
CPARS contractor past performance report card narratives and comparing that 
data to the information retrieved from these contracting professional interviews. 

F. SUMMARY 

This chapter presented an overview of the methodology for collecting the 
data, analyzing the CPARS narratives, and conducting interviews of DOD agency 
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contracting professionals. The objective scores were introduced and discussed. 
Next, the chapter reviewed the CPAR Quality Checklist to provide a framework 
for analyzing the database which is the basis for this research project. The 
methodology for the database statistical analysis was then explained. Finally, 
the chapter concluded with a review of interview questions posed to DOD 
contracting professionals. The next chapter discusses the results of the statistical 
analysis, responses to interview questions by DOD contracting professionals, 
and present findings that answer the research questions posed in Chapter III. 
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IV. ANALYSIS 


A. INTRODUCTION 

This chapter reveals the results of the analysis that examined the value of 
CPARS report card narratives for service contracts as they relate to their 
associated objective scores. This chapter begins with an overview of the data 
used to support the research presented in this report. The overview first 
describes the database created for this report in order to show from where the 
results of the statistical analysis were derived. The overview then provides a 
description of the series of interviews that were conducted to show from where 
the results of the interview findings came. Once the database and the interviews 
are described, the chapter explains the statistical analysis findings in detail. 
Following the description of the statistical analysis findings, this chapter 
describes the interview findings with each finding supported by specific 
commentary from the interviews. 

B. OVERVIEW OF THE DATA 

The database used for the statistical analysis portion of this project is 
limited to the 715 Army service contracts analyzed by a previous NFS MBA 
project (Hart et al., 2013). The selection criteria for those 715 contracts was 
limited to Army service contracts, “contracts only submitted by Mission 
Installation Contracting Commands (MICCs), and contracts from only five MICC 
offices” (from Hart et al., 2013). Of note, out of the 715 contract reports 
contained in the database, 22 were labeled as failures which “resulted in a total 
contract failure rate of 3.08%” (from Hart et al., 2013, p. 56). The determination 
for which contracts should be specified as failures “was determined by whether a 
contract received a marginal or unsatisfactory objective rating in quality of 
product/service, schedule, cost control, business relations, management of key 
personnel, or utilization of small business” (from Hart et al., 2013, p. 50). 
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Interviews with contracting professionals from two DOD contracting 
agencies were conducted and provided the basis for the interview analysis. 
These interviews consisted of a series of questions asking the subjects about 
how they use and to what extent they value CPARS and other sources of 
contractor past performance information and to what extent they value the 
narratives compared to the performance objective scores. Appendix D provides 
the specific questions asked during the interviews and the reasoning behind 
those questions. 

C. STATISTICAL ANALYSIS FINDINGS 

Next, we reveal the findings from our statistical analysis. First, we describe 
each of those findings in detail. We cover whether the narratives address all 
performance areas assessed, whether the narratives are based on objective 
data, whether the narratives are free of statements to avoid, whether the 
narratives are robust and comprehensive, and whether the narratives are written 
so that a contracting layman should understand the work performed. Then, we 
cover whether the narrative provides beneficial data above and beyond what 
could be gleaned from looking over the objective scores assigned. Lastly, we 
cover how well the narrative sentiment matches up with the objective scores 
assigned. At the end of this section, a summary chart of the statistical analysis 
findings is provided in Table 7. 

1. Narrative Addresses All Performance Areas Assessed? 

Overall, the narratives addressed all performance areas assessed 82% of 
the time. This means that -18% of the time, contracting professionals are not 
submitting to CPARS contractor performance narratives in accordance with the 
guidelines provided in the CPARS user’s manual for this requirement. This was 
less problematic with unsuccessful contracts at -95% than with successful 
contracts at -81%. The difference in the proportion of times that the narrative 
addressed all performance areas assessed in successful and unsuccessful 
contracts is statistically significant (p < .05). 
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2. Narrative Is Based on Objective Data? 

Overall, the narratives were based on objective data -77% of the time. This 
means that -23% of the time, contracting professionals were not submitting to 
CPARS contractor performance narratives in accordance with the guidelines 
provided in the CPARS user’s manual for this requirement. However, in 
unsuccessful contracts, the narratives were based on objective data 100% of the 
time. This is significantly different from the -77% in successful contracts (p < .01). 

3. Narrative Is Free of Statements to Avoid? 

Overall, the narratives were free of statements to avoid -97% of the time. 
This means that -3% of the time, contracting professionals were not submitting 
to CPARS contractor performance narratives in accordance with the guidelines 
provided in the CPARS user’s manual for this requirement. This was slightly 
more problematic with unsuccessful contracts at -86% than with successful 
contracts at -97% (p < .01). 

4. Narrative Is Robust and Comprehensive? 

Overall, the narratives were robust and comprehensive -63% of the time. 
This means that -37% of the time, contracting professionals were not submitting 
to CPARS contractor performance narratives in accordance with the guidelines 
provided in the CPARS user’s manual for this requirement. This was less 
problematic with unsuccessful contracts at -91% than with successful contracts 
at -62% (p< .01). 

5. Could a Layman Understand Work Performed? 

Overall, the narratives were written so that a contracting layman should 
understand the work performed -64% of the time. This means that -36% of the 
time, contracting professionals were not submitting to CPARS contractor 
performance narratives in accordance with the guidelines provided in the CPARS 
user’s manual for this requirement. This was less problematic with unsuccessful 
contracts at -82% than it is with successful contracts at -64% (p < .05). 
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6. 


Is the Narrative Beneficial Above and Beyond Objective 
Scores? 


As previously described in the methodology chapter, the chi-square test 
determines whether distributions of scores are statistically different. Using this 
test, we determined that there was a difference between successful and 
unsuccessful contracts in whether the narratives were beneficial above and 
beyond the objective scores. Unsuccessful contracts tended to have more 
beneficial CPAR report card narratives than successful contracts. 

Overall, the narrative provided an unsatisfactory amount of beneficial data 
to the user ~12% of the time. However, there were no unsuccessful contracts 
that provided an unsatisfactory amount of beneficial data. The narrative provided 
a marginal amount of beneficial data -22% of the time. There were no 
unsuccessful contracts that provided a marginal amount of beneficial data. The 
narrative provided a satisfactory amount of beneficial data -28% of the time. The 
narrative provided a very good amount of beneficial data -21% of the time. The 
narrative provided an exceptional amount of beneficial data -18% of the time. 
This was much more likely to occur with unsuccessful contracts than with 
successful contracts at -17%. 

7. Does the Narrative Correlate to Objective Scores Assigned? 

We again used the chi-square test and determined that there was not a 
difference between successful and unsuccessful contracts in whether the 
narrative correlates to the objective scores assigned. 

Overall, the narrative sentiment was contradictory to more than one of the 
objective scores assigned -2% of the time. The narrative sentiment was 
contradictory to one of the objective scores assigned -6% of the time. The 
narrative sentiment was satisfactory in describing accurately why the objective 
scores were assigned as they were -28% of the time. The narrative sentiment 
was very successful in describing accurately why the objective scores were 
assigned as they were -40% of the time. The narrative sentiment was 
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exceptionally successful in describing accurately why the objective scores were 
assigned as they were -24% of the time. 

Table 7 is a summary chart of the statistical database analysis that we 
have just described in detail. 



Result 

Overall 

Unsuccessful 

Contract 

Successful 

Contract 

Successful vs 

Unsuccessful 

Statistically 

Significant? 

P Value 

Narrative 

Addresses All 

Performance 

Areas Assessed? 

No 

18,46% 

4.55% 

18.90% 

Yes 

0.043 

Yes 

81.54% 

95.45% 

81.10% 


Namative Based 
On Objective 
Data? 

No 

22.52% 

0.00% 

23.23% 

Yes 

<.01 

Yes 

77.48% 

100.00% 

76.77% 


Namative Is Free 

Of Statements To 

Avoid? 

No 

3.08% 

13.64% 

2.74% 

Yes 

<.01 

Yes 

96.92% 

86.36% 

97.26% 


Namative Is 

Robust & 
Comprehensive? 

No 

37.48% 

9.09% 

38.38% 

Yes 

<.01 

Yes 

62.52% 

90.91% 

61.62% 


Could A Layman 

Understand Work 

Performed? 

No 

35.52% 

18.18% 

36.08% 

Yes 

0.042 

Yes 

64.48% 

81.82% 

63,92% 


Is The Narrative 

Beneficial Above 
& Beyond 
Objective Scores? 

Score 1 

11.61% 

0.00% 

12.12% 

Yes 

<.01 

Score 2 

21.68% 

0.00% 

22.37% 

Score 3 

27.97% 

18.18% 

28.28% 

Score 4 

20.84% 

27.27% 

20.63% 

Score 5 

17.76% 

54.55% 

16.59% 


Does The 

Narrative 

Correlate To 
Objective Scores 
Assigned? 

Score 1 

1.82% 

4.55% 

1.88% 

No 

0.141 

Score 2 

6.43% 

4.55% 

6.49% 

Score 3 

27.83% 

9.09% 

28.43% 

Score 4 

40.42% 

40.91% 

40.40% 

Score 5 

23.36% 

40.91% 

22.80% 


Table 7. Results of Statistical Database Analysis 
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D. INTERVIEW FINDINGS 

As previously discussed, we interviewed contracting professionals from 
two geographically diverse DOD contracting agencies. We did this to find out 
how they use and to what extent they value CPARS and other sources of 
contractor past performance information. We also wanted to find out to what 
extent they value the narratives compared to the objective scores. In the process 
of conducting these interviews, we made some interesting discoveries that could 
be useful in improving the contracting process specifically related to the narrative 
portion of past performance assessment reports. Below are the eight findings 
along with specific excerpts from the interviews supporting each finding. 

1. CPARS Is Still Often Not Reliable, Robust, or Comprehensive 
Enough 

The information in CPARS/PPIRS is oftentimes not reliable, robust, or 
comprehensive enough to allow source selection officials to place a significant 
enough weight on past performance. This finding was also noted in several GAO 
and OFPP memorandums mentioned earlier in the literature review chapter of 
this report. 

The Past Performance Information Retrieval System (PPIRS) is an 
information system that draws from the reports entered into CPARS. The 
contracting professionals we interviewed who deal regularly with utilizing the data 
in PPIRS felt that if the information in PPIRS were more reliable and consistent, 
they would be able to give more weight to it when making source selection 
decisions. The interviewees stated that when they knew there should be 
something in PPIRS on a particular contractor for a particular type of contract 
and dollar amount, they would often search for the contractor’s name in CPARS, 
only to receive this response: “no information has been entered.” 

Interviewee Number 2 made the recommendation that PPIRS would be a 
more useful tool if source selection teams could get more information in there 
from end users. Interviewee Number 2 was hesitant because this recommended 
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improvement will likely create more work for both the end user and the contract 
specialist, but suggested that DOD may want to lower the dollar value threshold 
to help contracting offices that normally deal with lower value contracts. 

Interviewee Number 5 stated that the CORs and contracting officers know 
generally how the report is supposed to be done, “but it is just something that is 
not executed as well as it could be.” Interviewee Number 5 stated that 
information from CPARS/PPIRS is not really helpful in source selection and 
contract administration because the database is not maintained well enough and 
not enough thought is put into them. Interviewee Number 5 stated. 

Mostly what I see is there is either satisfactory or excellent reviews 
even if there are problems, the problems are not always 
documented and I think that in one case at least it is actually the 
same CPARS pasted in to each year, each period and you could 
tell that even the dates were actually off sometimes. So it is not 
something that we put enough thought into, which makes it not a 
very useful tool in source selections, especially when you know 
other government agencies might be a little bit similar in their 
practices and it is not really something ... and it is something that 
you need to take with a grain of salt. I think when we are going 
through source selection really what we look for is the offeror’s 
technical understanding of the requirements. Sometimes it is 
obvious that there is a little bit more business development 
language than actual understanding of the work and that seems to 
be more of the focus than the actual CPARS. 

We also noted similar issues in conducting our database analysis, that in some 
cases the exact or almost the exact same narrative was cut and pasted into 
several different contract report cards. 

2. Unsuccessful Contracts Are More Reliable, Robust, and 
Comprehensive 

Unsuccessful contracts tend to have more reliable, robust, and 
comprehensive past performance information available in their CPARS/PPIRS 
reports. 
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Interviewee Number 3 stated that there might be a reason why the 
unsuccessful contracts were typically more robust and comprehensive than the 
successful ones: 

If the contractor is really just coasting along doing everything really, 
really well, I think a lot of times you might not have a really in depth 
report because the COR says, “hey, everything’s good. I don’t have 
to worry about anything.” So, they’re just going to kind of mark it 
up, “hey, they’re good or they’re bad or whatever on everything.” I 
think the ones that maybe would take longer is when you have 
issues. 

Interviewee Number 3 went on further to state that narratives might be longer for 
a poor performance because 

they know that those are going to get ... the contractor is going to 
be able to review those as well as the contracting officer, so the 
COR should put more time into those to just, you know, really 
explain himself why you’re giving a poor rating. Typically, if you’re 
getting a good rating, the contractor is just going to say, “yup, I 
agree. Go ahead and post it to PPIRS. I agree with it.” But if 
you’re giving them a negative rating, you’re really going to want to 
explain yourself. 

3. Assigning Weight to Past Performance Using the Information 
Avaiiabie 

The appropriate amount of weight that should be assigned to past 
performance in making source selection decisions should be correlated to the 
source, availability, quality, and relevancy of the past performance information. 

Interviewee Number 3 made the comment that the weight assigned to 
contractor past performance in PPIRS is basically a direct correlation to the 
relevancy of the past performance combined with how robust and comprehensive 
the written narrative is. 

Interviewee Number 5 stated that if they had negative past performance 
information from PPIRS or from a questionnaire, they would give it more weight, 
but generally they have to give more weight to the technical portions of a 
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proposal because PPIRS is not able to produce the information they need and 
the questionnaire is not likely to produce negative past performance information. 

Interviewee Number 6 stated that it is more beneficial if the source 
selection authority is able to access multiple contracts of similar scope and 
complexity. Interviewee Number 6 stated, “If there is consistency I would place 
greater value in that evaluation. If they are all over the place then it is really hard 
to put a ... it is kind of subjective how much value you put into a particular 
CPAR.” 


4. What to Do With Contradictory Past Performance Information 

PPIRS sometimes contains information in the narrative that is either 
contradictory or does not quite match up with the objective scores. When the 
objective scores and narrative sentiment in PPIRS is mismatched, contracting 
professionals tend to give more weight to the narrative versus the objective 
scores. 

Interviewee Number 1 made the strong recommendation that the 
contracting professionals who regularly deal with PPIRS in looking for past 
performance information need to take note of when there is conflicting 
information between the objective scores assigned and the written narrative. That 
interviewee said that when source selection team members find something like 
that, they should send the report back to whoever entered the information and 
make them correct that section. This comment was reiterated by Interviewee 
Number 4: 

I’d send it back and tell whoever wrote it to come up with why their 
scores was what it is and explain to them, look. I’m looking at this 
and I’m reviewing what you wrote, but it doesn’t justify the 
outstanding rating or the poor rating. 

Interviewee Number 2 stated that they give considerably more weight to 
the narrative versus the objective scores because 

sometimes you will see the government rated them very good, but 
then the description you’re reading is like ... it doesn’t match ... 
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there’s a description next to what very good should be and then 
what they’re saying doesn’t match very good. 


Interviewee Number 2 also stated that the narrative is usually more beneficial 
because sometimes 

there’s a bunch of different categories and depending on what the 
contract is for, a lot of them may be N/A, so the description is a lot 
better than just saying like ... like if you’re doing a firmed fixed price 
contract but for some reason they rate cost control and they say, 
“excellent,” well, I would hope so because we’re only paying the 
firm fixed price. 

5. Reliable and Accurate Past Performance Retrieval Methods 
Are Needed 

Contractor past performance reports might not be as accessible as they 
should be in PPIRS because contracting professionals are not always applying 
due diligence in identifying the appropriate entity using identifying information 
within the contractor’s parent organization. Examples of such identifying 
information would be an appropriate cage code or DUNS (Dun and Bradstreet 
Data Universal Numbering System) number. 

Interviewee Number 3 alluded to the fact that it might be more beneficial to 
have the PPIRS database set up to more easily find similar contracts for a given 
contractor. Interviewee Number 3 stated that 

one of the things we’re really looking for when we’re doing our 
source selection is the relevancy of past performance, so not just, 

“hey, they performed well on a bunch of contracts.” That matters, 
but not as much as it would matter if they performed really well on a 
similar type contract to the one that we’re looking to award. So, 
you know, you may have a contract for a completely different 
service that this company did and they got really awesome ratings, 
that gives us an idea that they’re able to perform well and are 
probably good workers, but what does that really tell us about how 
they’re going to perform this contract that is a completely different 
type of service. So, we use it, but I would say we’re going to ... we 
have other methods of doing past performance research to look at 
their relevancy and, you know ... yeah, their relevancy to the 
contract that we’re looking to award. To me that’s more important. 

It’s how will they perform on a similar type contract. But if they’ve 
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never done this type of work before, yeah, then it would be weighed 
more heavily than this stuff that you’re looking at in PPIRS because 
that’s all you really have to go on. So, I think the reports are good 
enough to explain, hey, this is the type of work that’s being done, 
so you can really make a good comparison. 

We noted that while this might give the contracting professionals 
information to use in weighing past performance, the information provided as 
references from contractors would almost certainly be skewed to positive 
references. A contractor will most likely omit a recent contract reference that 
they know they performed poorly on. This is really where the process of 
incorporating past performance information into source selection is failing in that 
PPIRS should be the avenue for source selection officials to find negative past 
performance information if it exists. Too often, it looks like data in the system is 
not organized well enough. The way the data is organized is not helping source 
selection officials mine the database to uncover this recent and relevant negative 
past performance information as well as it should. 

When asked about ideas of how CPARS/PPIRS could be improved. 
Interviewee Number 9 had the recommendation that “there could be something in 
the PPIRS system that would allow you ... to weed through the relevancy 
aspect.” 

6. Past Performance Questionnaires Are Inadequate 

Source selection officials often need to solicit contractors for references or 
ask them to fill out a past performance questionnaire because there is a lack of 
reliable, robust, and comprehensive past performance information available in 
PPIRS. 

Because they are often unable to find relevant, robust, and 
comprehensive information about a particular proposal in PPIRS, Interviewee 
Number 3 stated. 

In our solicitations or our RFPs, when we put those out then we’ll 
ask for the contractor to submit ... for example, we might ask them 
to submit five of their most recent contracts that they’ve done for 
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similar type work with references. So, they would provide 
references of, you know, John Doe who they did work for in, you 
know, for NAVSEA out of Washington, DC, or something like that. I 
could contact him and say, “hey, how did these guys perform on 
this?” Or, I could send them an actual ... like a survey to fill out and 
then we’ll get that all filled out and then we know that it’s a relevant 
contract, we know it’s a recent contract that they’ve done. So, it’s a 
lot easier to use that information than it is a lot of times in PPIRS, 
because if you go into PPIRS it’s like the most recent one might be 
like 2009 or 2010 and it’s like a completely different contract. 

Interviewee Number 5 made note of a similar questionnaire and reiterated 
the issue with it by stating. 

When we have source selections here at our agency, we usually 
have the contractors fill out a past performance questionnaire. It is 
a standard questionnaire that we send out for each offeror and a 
government representative has to fill them out and send them back 
directly to the contracting officer. Now those are a little bit different 
in that it is not CPARS, but they do have to reference it and we do 
validate that it is accurate. But I mean I think that sometimes you 
find that of course they are only going to give references for their 
best clients so they are always going to be good or at least 
satisfactory. 

7. Timely Reports Must Be of High Quality to Be Useful 

There has been a recent increase in senior-level interest to complete 
required CPARS reports in a timely manner but not an increased interest in the 
quality of those reports. 

Interviewee Number 2 stated that the narratives are 

not as lengthy as they could be. Like, if you figure we spend three 
months on something, you would think that they would have like a 
real good paragraph ... very detailed—but sometimes it seems like 
I get more information talking to people than the two sentences 
they put in CPARS. 

Interviewee Number 7 brought up a noticeably increased emphasis on 
timely completion of the report card but said it was just as important to have 

quality, because really you are wasting your time if you are just 
putting this ... you are wasting everybody’s time. The emphasis is. 
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right now more timeliness, we have gotten it from very high up that 
they want these done, they haven’t really said done well. So I think 
after we get our hands around and get people doing it within the 
timeframe they are supposed to be doing it, then we would like to 
turn to the quality of them. I know some of the field activities have 
already taken that step. I heard one field activity actually had ... 
the commanding officer went through every single CPAR done in 
the last year and pulled some of them back and said that he wanted 
it redone because it didn’t have a narrative that supported the 
ratings they gave them. So with that, if you are getting good, 
quality CPARS, I think it is a very valuable tool and one that should 
be used in source selections because it is like having—it is like the 
Angie’s List for the federal government that you have something 
that you have feedback on customers. 

8. Miscellaneous Recommendations for Improvement 

Several of the interviewees made recommendations for improving 
services acquisition related to contractor past performance information. The 
recommendations were to add more analysis tools and performance metrics in 
PPIRS, better monitor the workload of CORs, better train the workforce on writing 
CPARS narratives, and better broadcast the findings of CPARS Program Office 
audits. 


Interviewee Number 6 recommended that the program office could 

load CPARS up. I mean, you could have cost analyses, you could 
have trend analyses, and you could have performance analyses. I 
mean did they perform, how well did they perform, did they make 
their timelines, was their product accepted, did it go through 
developmental tests, operational tests, was it kicked back? 

Interviewee Number 7 mentioned the workload of the CORs as a potential 
factor in the inadequacy of the contract report cards, stating, 

I did statistics on how many contracts and we had some CORs that 
were definitely way overloaded. Based on what they were saying, 
really to do a good job on a good contract you should only be a 
COR on like one major contract. Smaller ones, two to three, but we 
have people—six or eight, more than that. So our workload was—I 
don’t think that was the reason they weren’t doing their CPARS, but 
it was one of the reasons they weren’t doing all of the things they 
needed to do. 
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When asked about ways that would help improve CPARS so that it could 
be more useful, Interviewee Number 8 stated, “Training is probably the biggest 
one.” Interviewee Number 8 also stated, “We don’t train our people very well in 
writing, especially when you have engineers that are filling out some of these 
things. They can be rather cryptic.” 

Interviewee Number 8 also made the recommendation that when there are 
audits conducted at a command regarding CPARS, it needs to be better 
broadcast to the people involved in services acquisition to better gain access to 
the results of those audits along with the trends across the whole of government. 
Specifically, Interviewee Number 8 stated, 

the CPARS Program Office goes out and does audits periodically. 

There really hasn’t been—one time they did a lessons learned and 
publicized that, but it would be good to see some trends, things that 
they could suggest that they are seeing people having a hard time 
doing. You know, that would help improve CPARS as a role if they 
are finding some people; certain contracts are harder to do. I don’t 
know. Whatever they are finding in their audits, we never get the 
result. 

E. SUMMARY 

This chapter revealed the results of the research that examines the value 
of CPARS report card narratives for service contracts as they relate to their 
associated objective scores. This chapter began with an overview of the data 
used to support the research presented in this report. The overview first 
described the database created for this report in order to show from where the 
results of the statistical analysis originated. The overview then provided a 
description of the series of interviews that were conducted to show from where 
the results of the interview findings came. Once the database and the interviews 
were described, the chapter explained the statistical analysis findings in detail. 
Following the statistical analysis findings, this chapter described the interview 
findings, with each finding supported by specific commentary from the interviews. 
The next chapter includes a summary of the research, conclusion of the findings, 
and areas for further research to enhance the knowledge of service contracting. 
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V. SUMMARY, CONCLUSION, AND AREAS FOR FURTHER 

RESEARCH 


A. SUMMARY 

The DOD obligated approximately $360 billion in fiscal year 2012 for 
contracts for supplies and services (GAO, 2013). In order to support best 
practices for government acquisition, the Office of Federal Procurement Policy 
(OFPP) directs in FAR Part 42 that federal contracting agencies shall use 
CPARS as their primary tool for documenting contractor past performance. 

CPARS is a system that was designed to be used by contracting 
professionals for submitting and retrieving contractor past performance 
information. There have been many discrepancies and failures specified in 
recent GAO reports with the utilization of CPARS. Government contracting 
agencies are failing to properly document contractor past performance 
information within the schedule requirements mandated by OFPP. In a 2009 
report, the GAO analyzed data from 2007 and showed that DOD contracting 
components completed required contractor past performance report cards less 
than half of the time (GAO, 2009). 

Subsequently, the DOD increased its focus on training and education for 
contracting professionals, which has led to an increase in contractor performance 
assessments being completed and submitted to PPIRS. In the last published 
report during the research period of this project, the GAO (2013) noted significant 
gains in completion rates. Fifty-six percent of required reports were completed in 
2011 while 74% were completed in 2013. However, according to the same 
report, over half of these reports were submitted late. In addition, many CPARS 
report cards contain narratives that are either insufficiently detailed or conflict 
with their associated objective scores. Late reports lacking sufficient accurate 
information provide less-than-optimal information to the contracting professionals 
who rely on these report cards for source selection and contract administration 
purposes. 
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B. CONCLUSION 


1. Research Findings 

The purpose of this research was to determine the value of contractor 
performance assessment report narratives for services contracts by comparing 
the relationships between narratives and objective scores in order to recommend 
improvements to the CPARS contractor performance information documentation 
process. To identify this value, the research focused on answering five research 
questions. Our research questions and findings are as follows. 

• To what degree are government contracting professionals 
submitting contractor performance narratives to CPARS in 
accordance with the guidelines provided in the CPARS user’s 
manual? 

To answer the first research question, the data indicated that, overall, contracting 
professionals inputting reports into CPARS are doing a highly effective job at 
ensuring that their narratives are free of statements to avoid (-96.9%). These 
professionals are not doing as well in addressing all performance areas 
assessed in their objective scores (-81.5%) or in ensuring that the narratives are 
based on objective data (-77.5%). The areas that contracting professionals seem 
to have the most issues with, however, are writing a comprehensive narrative 
(-62.5%) and ensuring that a contracting layman (such as someone who might 
need to access CPARS data in order to make a decision about a future contract 
award) can fully understand the work performed (-64.5%). With the exception of 
the narratives being free of statements to avoid, contracting professionals were 
more effective in all categories when writing narratives for unsuccessful contracts 
than with successful ones. 

• What is the added value of the contractor performance narratives 
beyond the value of the objective scores for performance? 

In answering the second research question, the data indicated that, 
overall, contracting professionals inputting data into CPARS are doing a better 
job at providing beneficial data in the narrative when the contract is unsuccessful 
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versus when it is successful. Only 38.6% of the sample narratives, whether 
successful or unsuccessful, provided a very good or exceptional amount of 
beneficial data above and beyond what could be gleaned from looking over the 
objective scores assigned. This shows that there is clearly room for improvement 
in this area. 

• What is the statistical relationship between the sentiment contained 
in the narratives and the objective scores for contractor 
evaluations? 

In answering the third research question, the data indicated that, overall, 
contracting professionals who input data into CPARS are writing information in 
their narratives that contradicts at least one of the objective scores assigned 
~8.3% of the time. This leaves room for improvement. Contracting professionals 
were slightly better at matching the narrative sentiment to the objective scores in 
unsuccessful contracts (~81.8% scoring either very good or exceptional) than in 
successful contracts (-63.2% scoring either very good or exceptional). 

• To what degree do the interview findings contradict, support, or 
enhance the findings for the three research questions above? 

In answering the fourth research question, the interviews indicated that 
there are no issues with narrative statements to avoid. This supports the 
database analysis conclusion that contracting professionals inputting reports into 
CPARS are highly effective at ensuring that their narratives are free of 
statements to avoid (-96.9%). The interviews indicated that there are only a few 
minor issues with addressing all performance areas assessed and ensuring that 
the narratives are based on objective data. This supports the database analysis 
conclusion that contracting professionals are doing a decent job in addressing all 
performance areas assessed in their objective scores (-81.5%) or in ensuring 
that the narratives are based on objective data (-77.5%). The interviews 
indicated that there are significant issues with writing comprehensive narratives. 
This enhances the database analysis conclusion that contracting professionals 
seem to have significant problems with writing comprehensive narratives 
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(~62.5%). The interviews indicated that there are some issues with writing 
narratives that ensure a contracting layman can understand the work performed, 
especially with engineering-specific entries. This supports the database analysis 
conclusion that contracting professionals seem to have significant problems with 
ensuring that a contracting layman (such as someone who might need to access 
CPARS data in order to make a decision about a future contract award) can fully 
understand the work performed (-64.5%). 

• What conclusions or recommendations can we draw from the 
answers to the research questions above? 

Lastly, in answering the fifth research question, we can conclude that 
contracting professionals should get some additional/remedial training on writing 
robust and comprehensive narratives that the contracting professionals who are 
accessing these reports are able to fully understand. It may also be beneficial for 
contracting professionals to get additional training in ensuring that the narratives 
address all performance areas assessed and in ensuring that the narratives are 
based on objective data. With the issues outlined in this section, we can also 
recommend that quality control for those approving the narratives posted to 
CPARS, as well as those using the data from PPIRS, can be improved by 
sending unacceptable reports back to the originator for correction. We conclude 
that there is no further emphasis or training needed in ensuring that the 
narratives are free of statements to avoid since the data does not show that this 
is a significant issue. 

2. Additional Findings 

In the process of completing our research, especially in conducting our 
interviews, we found several other pieces of information that could be beneficial 
to the acquisition community with regard to services acquisition that we would 
like to present here. 

Finding 1: The information in CPARS/PPIRS is oftentimes not reliable, 
robust, or comprehensive enough to allow source selection officials to place a 
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significant enough weight on past performance. This finding was also noted in 
several GAO and OFPP memorandums mentioned earlier in the literature review 
chapter of this report. Without reliable, robust, and comprehensive information, it 
is not possible to properly utilize CPARS/PPIRS in making fair and accurate 
source selection decisions based on past performance. 

Finding 2: Unsuccessful contracts tend to have more reliable, robust, and 
comprehensive past performance information available in their CPARS/PPIRS 
reports. Source selection decisions should be based on reliable, robust, and 
comprehensive past performance information for both successful and 
unsuccessful contracts in order to make fair and accurate award decisions based 
on past performance. 

Finding 3: The appropriate amount of weight that should be assigned to 
past performance in making a source selection should be correlated to the 
source, availability, quality, and relevancy of the past performance information. 
Implementing this finding will be difficult because the task of assigning weight to 
past performance information is normally completed during the acquisition 
planning phase of the acquisition process. Contractor past performance is 
normally mined later in the process after the acquisition team has received 
proposals. 

Finding 4: PPIRS sometimes contains information in the narratives that is 
either contradictory or does not quite match up with the objective scores. When 
the objective scores and narrative sentiment in PPIRS is mismatched, 
contracting professionals tend to give more weight to the narrative versus the 
objective scores. When the objective scores and narrative sentiment in PPIRS 
are mismatched, every effort should be made to send unacceptable reports back 
to the originator for correction. If this is not possible, the weight assigned to past 
performance in the source selection decision should be reduced to reflect the fact 
that there is a contradiction in the report, thereby reducing its reliability. 
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Finding 5: Contractor past performance reports might not be as 
accessible as they should be in PPIRS because contracting professionals are not 
always applying due diligence in identifying the appropriate entity using 
identifying information within the contractor’s parent organization. Examples of 
such identifying information would be an appropriate cage code or DUNS (Dun 
and Bradstreet Data Universal Numbering System) number. It is imperative that 
source selection officials are able to access the correct reports for a particular 
contractor’s past performance if they exist. There must be a uniform way of 
inputting and accessing the correct information in PPIRS if source selection 
officials are to make fair and reliable award decisions using the past performance 
information that we have about a particular contractor. 

Finding 6: Source selection officials often need to solicit contractors for 
references or ask them to fill out a past performance questionnaire because 
there is a lack of reliable, robust, and comprehensive past performance 
information available in PPIRS. The fundamental issue with using past 
performance information about a contractor that is provided by that contractor is 
that it will almost certainly only include positive references and omit references 
where the contractor did not perform well. When this is the only past performance 
information available to source selection officials, the weight assigned to past 
performance in the evaluation criteria needs to be reduced to reflect the 
reduction in the reliability and comprehensiveness of the information. 

Finding 7: There has been a recent increase in senior-level interest to 
complete required CPARS reports in a timely manner but not an increased 
interest in the quality of those reports. The focus needs to be on both the 
timeliness and quality of the reports. Without quality reports, the information in 
the report will not be able to be assigned much weight in future source selection 
decisions and therefore will just be a waste of time for everyone involved in the 
process. 

Finding 8: Several of the interviewees made recommendations for 
improving services acquisition related to contractor past performance 
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information. The recommendations were to add more analysis tools and 
performance metrics in PPIRS, better monitor the workload of CORs, better train 
the workforce on writing CPARS narratives, and better broadcast the findings of 
CPARS Program Office audits. With improved analysis tools, appropriate 
workloads for contracting professionals inputting the past performance into 
CPARS, and a more robust avenue for correcting workforce problem areas, the 
past performance information in PPIRS will undoubtedly improve and better 
assist source selection officials in making fair and reliable award decisions. 

3. Recommendations 

Based on our conclusions, we identified the following eight 
recommendations. 

Recommendation 1: Remedial training should be implemented for all 
members of the acquisition teams that input and utilize past performance 
information in CPARS and PPIRS. Training should focus on the following areas 
related to writing comprehensive narratives: ensuring that acquisition team 
members (i.e., someone who might need to access CPARS data in order to 
make a decision about a future contract award) can fully understand the work 
performed, addressing all performance areas assessed in their objective scores, 
and ensuring that the narratives are based on objective data. 

Recommendation 2: There needs to be an overall push for higher quality 
past performance report submissions in CPARS/PPIRS to allow acquisition 
teams to assign weight to past performance in source selection decisions without 
having to worry about the source, availability, quality, and relevancy of the past 
performance information. 

Recommendation 3: An analysis should be performed on the 
CPARS/PPIRS database by the program office to ensure that contracting 
professionals are in fact able to accurately input and retrieve the past 
performance information on the correct contractor 100% of the time that the 
information is available. 


65 



Recommendation 4: A review of the CPARS/PPIRS database by the 
program office should be conducted to see whether additional analysis tools can 
be added to better assist contracting professionals in identifying past 
performance trends for a particular contractor. 

Recommendation 5: Senior officials at contracting commands should 
emphasize not only the timeliness of CPARS submissions but also emphasize 
the quality of those submissions. 

Recommendation 6: A requirement should be added to the CPARS 
report card approval and posting process that allows for customer feedback on 
contractor performance. Agency approving officials should be encouraged or 
required to collect (either directly or via the COR) and consider input from the 
customer regarding contractor performance. This will encourage the submission 
of more accurate and robust CPARS report cards. 

Recommendation 7: A review of the workload of CORs should be 
conducted at contracting commands to ensure they have the appropriate amount 
of time to dedicate to the proper submission of past performance information. 

Recommendation 8: the CPARS Program Office should look at ways to 
better broadcast the findings of their audits to help the workforce improve over time. 

C. AREAS FOR FURTHER RESEARCH 

Additional research would be valuable in several areas to further 
determine the value of the CPAR narratives for the acquisition process. 
Obtaining a larger, more diverse, and more recent sample database and redoing 
the database analysis would add significant insight into how the contract report 
cards for other types of contracts and for different uniformed services are 
impacting the acquisition process. Specifically, obtaining a new database that 
contains contracts for supplies and contracts for different military branches, and 
comparing CPARS report cards from different agencies would allow for some 
interesting comparisons that could yield significantly different recommendations 
for improving the use of past performance information in the acquisition process. 
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APPENDIX A. RESULTS OF THE INTER-RATER RELIABILITY 

TESTING 



Rater & 
Score 


Rater #1 


Rater #2 


Rater #3 


Score 


Narrative 

Addresses 

All 

Performance 

Areas 

Assessed 

(Yes=1, 

No=0) 


Narrative 
Based On 
Objective 
Data 
(Yes=1, 
No=0) 


Narrative is 
Free Of 
Statements 
To Avoid 
(Yes=1, 
No=0) 


Narrative is 
Robust & 
Comprehensive 
(Yes=1, No=0) 
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APPENDIX B. SPECIFIC EXAMPLES OF NARRATIVE 
STATEMENTS TO AVOID 


FROM THE CPAR QUALITY CHECKLIST _ 

Block 20: Sample Narrative Statements to Avoid 

The Contractor’s performance in this area was exemplary. They were proactive in satisfying Electrical Kit 
Product Performance requirements. They produced a superior product for the customer. In many 
instances, they performed engineering tasks outside the scope of the contract. 

“Outside the scope of the contract” - This phrase should not be in a CPAR narrative. It implies that the 
Contractor performed work not legally required and is eligible for an equitable adjustment to the contract. An 
equitable adjustment means that the program office/customer will have to come up with additional funds to 
pay for the additional tasks. 

In our opinion, the Contractor’s performance in the systems engineering area was very poor. Kit hardware 
deficiencies were observed and it appeared that the Contractor lacked systems engineering knowledge and 
expertise. We believe that some of the contractual kit requirements will not be met. It is our hope that 
additional factory testing will eliminate these hardware deficiencies. If management had responded in a 
timely manner, the requirement might have been satisfied. Additionally, we were not happy with the initial 
factory testing, and did not like their “fly and fix” philosophy of testing. 

“In our opinion” - This is a subjective phrase which gives the impression that there is no firm evidence to 
prove poor performance. 

“Appeared” - This is a speculative remark which does not prove that they lacked systems engineering 
knowledge. 

“We believe” - This is also a speculative remark. It does not prove that they did not satisfy some kit 
requirements. 

“It is our hope” - This statement does not belong in a CPAR narrative. The issue is whether the Contractor 
will correct the deficiencies using factory testing. If so, the narrative should indicate the pending corrections. 
If not, justification should be provided as to why the factory testing failed to correct the problems. 

“We were not happy” - This is an emotional and subjective statement which should be avoided. The CPAR 
should reflect justification for the successes/failures from the factory test. 

“We did not like” - The customer should evaluate the results of the fly and fix tests in detail, not their testing 
technique. 

The Contractor was late in delivering all of the 100 electric kits. We think that one reason is that their 
systems engineering effort was poor due to several electrical component deficiencies. Another reason 
could be that their ability to manage the electrical subcontracts left much to be desired. We established a 6 
month extension to the contract. We hope they can deliver the 100 kits without significant discrepancies. 

“We think” - This phrase implies that the customer has not proven the Contractor’s poor performance with 
evidence. 

“Could be” - This phrase indicates that the customer is not sure that the reason for the deficiencies is poor 
management. There is no proof of poor management here. 

“We hope” - This phrase implies that the delivery of the kits without deficiencies in the time period allotted is 
a desire, not a contractual requirement. 
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APPENDIX C. AN EXAMPLE OF AN INSUFFICIENT NARRATIVE 
VERSUS. A BETTER WAY TO WRITE THIS 


TAKEN FROM THE CPAR QUALITY CHECKLIST _ 

Block 20: Sample CPAR Narrative 

Insufficient Narrative: 

Schedule: Very Good - In our opinion, the Contractor has done really well in terms of 
schedule. The Training Manager, Jack Jones, is pleasant and easy to work with. He 
adapts to our schedule changes amazingly and never complains. He also went above 
and beyond and fixed our printer and fax without charging the Government and he 
continued to meet all the contract objectives in the interim. Great job! 

The example above is missing detail to support the score and supporting documentation 
and metrics. In addition, it uses an individual’s name, addresses work outside the 
contract scope, and uses subjective phrases. 

A Better Way to Write This: 

Schedule: Very Good - Contractor successfully executed the delivery and training 
requirements for this period ahead of schedule. For example, there were 20 training site 
visits scheduled for this period however, the Contractor conducted 31 visits in the same 
period of time. The Contractor also met 100% of the 13 contract data requirements in a 
45 day timeframe versus the 60 days allotted. This resulted in data requirements 14-20 
being completed earlier than anticipated. This was done with minimal supervision by the 
program office hence allowing more time for additional projects. A 20 site preventative 
maintenance visit ran behind schedule for the first 8 months of the reporting period due to 
equipment failures, but Contractor management was able to bring the visit back on 
schedule due to implementation of an aggressive quality management system and 
spares availability policy. 
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APPENDIX D. INTERVIEW QUESTIONS FOR DOD 
CONTRACTING PROFESSIONALS 

INTERVIEW INTRODUCTION 

The following section will be read verbatim to each interview subject in 
order to explain to them the purpose of the interview and to ask their permission 
to perform the interview. 

Project Title: 

Determining the Value of Performance Evaluation Narratives for Service 
Contracts 

Purpose of Interview: 

We are three graduate students enrolled in the Acquisition and Contract 
Management Curriculum at the Naval Postgraduate School. After graduating, we 
will be working in the contracting field for the Department of Defense. We are 
conducting research in an effort to determine the value of contractor performance 
assessment report narratives for services contracts. In Department of Defense, 
members of the acquisition team submit contractor past performance information 
to Contractor Performance Assessment Reporting System (CPARS). This 
information includes objective (numeric) scores for contractor performance in five 
categories as well as a written narrative. Contracting professionals and source 
selection team members are able to retrieve contractor past performance 
information that was submitted to CPARS through the Past Performance 
Information Retrieval System (PPIRS). 

We have studied and conducted statistical analysis on CPARS contractor 
past performance narratives and how they relate to their associated objective 
scores. As part of this research, we would like to ask for your assistance by 
allowing us to interview you. The purpose of the interview is to obtain data about 
how and to what extent you use the objective scores and narratives when 
reviewing CPARS (or PPIRS) report cards for source selection and contract 
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administration. This interview data, combined with a statistical analysis of 
narratives and objectives scores, should allow us to make recommendations for 
improving services acquisition and streamlining the way contracting professionals 
submit to and utilize these systems. Our goal is to make these programs more 
useful to you, the contracting professional, as well as easier to use, if there is any 
improvement to be had. 

Your command and department head have invited us here, upon our 
request, to conduct this interview and you have been selected by them to be a 
potential interviewee based on your role and experience as a contracting 
professional. Participation in the interview is voluntary and anonymous. Your 
responses to our questions will not be able to be traced back to you or your 
command and they will not be reported to your supervisors or chain-of-command. 
This interview consists of 13 questions and should take about 30 minutes to 
complete. 

Do you agree to assist us in conducting this research by allowing us to 
interview you? 

Goal of the Interview: 

The previous section Purpose of Interview was formuiated with the 
foliowing goals in mind: 

• Brevity . We do not want the subjects to lose interest before the 
interview begins. 

• Clarity . This introduction to our research is concise and detailed 
enough to be readily understood by a contracting professional with 
enough knowledge of the subject matter to be useful in our 
research. 

• Unbiased . We do not want to show whether we believe contractor 
past performance objective scores have any more or less value 
than the associated narratives. We do not want to specify whether 
there is in fact any way to improve upon the CPARS/PPIRS 
programs. In an effort to avoid encouraging a self-serving bias in 
interview subjects we do not want to specify that submitting 
contractor past performance information to CPARS is directed by 
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Department of Defense. We want to get them to answer questions 
honestly without concern that they might not have been closely 
following protocol. 

• Encouraging . We want to identify with our subjects as fellow 
government contracting professionals which a mutual interest in 
improving our processes and systems. We want to encourage 
them to agree to the interview and to be forthcoming because in 
doing so they might help all of us in the future. 

• Non-retribution . Subjects should be clear on the fact that there will 
be no way that this interview can come back to hurt them and that 
their identities will be completely protected. Interviews are 
completely voluntary and encouraged by the interviewee’s 
command. 

INTERVIEW QUESTIONS 

How often do you use contractor past performance information for 
source seiections? 

[Provide Verbal Categories of Options for Answers] 

This question allows us to approximate the level of depth of familiarity the 
subject has with contractor past performance information in general and 
specifically whether he or she participates in contract source selections. This 
question is carefully worded in order to avoid asking alternative questions such 
as: How often do you access PPIRS? What portion of the time, when 
conducting source selection, do you use contractor past performance 
information? We want to know specifically how many times in a given period 
they use contractor past performance information for source selection in order to 
determine the subject’s depth of experience and to determine to what extent 
contractor past performance information is used as compared to how frequently it 
is used during contract administration. 

In order to help the subject understand the question we will provide 
examples of potential answers, carefully in order to avoid injecting bias, such as: 
once a year, or several times a day. Using examples on the extreme ends of the 
range will keep from planting a likely answer in the mind of the subject. 
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How often do you use contractor past performance information for 
contract administration? 

[Provide Verbal Categories of Options for Answers] 

This question is the same as the previous question but for contract 
administration instead of source selection. The purposes of this question are the 
same as with the previous question except for isolating contract administration 
from source selection. We do not believe it would be sufficient to ask one 
question: How often do you use contractor past performance information for 
contract administration and/or source selection? For the purposes of the 
research we want to know to what extent it is used for each specific purpose. 

When inputting past performance data into CPARS, what is the 
typicai amount of time spent per contract gathering information and 
inputting it into the system? 

There is some purposeful ambiguity in this question. Some contracting 
professionals write a the entire narrative for one CPARS report card while in 
other cases, especially for large and complex acquisition, the task is shared by 
up to several members of the acquisition team. We want to allow the interview 
subject to specify how much time is spent per contract gathering and inputting 
information whether or not all of the effort was conducted by the interviewee. 
This may require that the subject make some estimates of how much time other 
individuals are spending on this task. 

Does this amount of time vary greatiy or is it usuaiiy around the 
typicai amount of time? 

The purpose of this question is to establish whether there is a lot of 
variability in the amount of time one spends putting together a CPARS 
submission. If there is not much variability then the follow up question would be 
to determine whether there is much variability in the complexity of the contracts 
the subject handles. If there is a lot of variability, we want to determine whether 
the variability is based primarily on the size and complexity contracts. We clarify 
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the impact of the data from the previous question with one of the two foiiowing 
questions. 

[ask the following question if subject responded with answer more 
similar to “the amount of time is usually around the typical amount of 
time”] 

Are the contracts you write or administer usually relatively similar in 
size and complexity or do they tend to vary greatly? 

[ask the following question if subject responded with answer more 
similar to “the amount of time varies greatly”] 

Do you typically find that gathering and submitting information to 
CPARS takes more time for larger more complex contracts or do you find 
that the amount of time varies not related to the size and complexity of the 
contract? 

Subjects wiil answer one of the two previous questions. 

Typically, what portion of the total time it takes to prepare and 
submit a CPARS report is spent preparing and writing the narrative portion 
of it? 

The purpose of this question is to determine what portion of their time is 
spent preparing the narrative which will be one of the evaluation factors 
considered when determining the relative value of the narrative. Our evaluation 
will compare the relative value of the narratives to the relative amount of effort it 
takes to create them. 

What is the typical number of people involved per contract in 
entering contractor past performance information into CPARS? 

The purpose of this question is to tell us the typical size of the acquisition 
team. This provides us data useful in analyzing the CPARS written narratives. 
Some narratives are very uniform throughout their sections in terms of positive or 
negative sentiment, while others have much more variation. Some CPARS 
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written narratives are written by one individual while others are written in parts by 
several. Useful data will show whether it is much more or much less likely that 
an individual written narrative was written by one person than more than one 
person. 

[Ask the following question If the answer to the previous question 
was greater than one] 

Which members of the acquisition team (contracting officer, COR, 
program manager, requirements manager, etc.) are responsibie for entering 
past performance information into CPARS? 

The purpose of this question is to determine specifically which members of 
the acquisition team are most likely to participate in the writing of a CPARS 
narrative. 

When you are conducting a source seiection or administering a 
contract does contractor past performance information have any impact? 

The purpose of this question is to determine to what extent, in general, the 
interview subject values contractor past performance information. In this case, 
we feel that combining source selection and contract administration will provide 
adequately specific and useful data for our research purposes. The question is 
specifically worded to avoid injecting bias and will apply to all interview subjects 
irrespective of answers to previous interview questions. If the interview subject 
answers in the negative then there is no need for a follow-on question. 

[If answer to previous question was in the affirmative, ask the 
foiiowing question] 


To what extent does contractor past performance information impact 
your source seiection or contract administration? 

The purpose of this question is to determine the value of CPARS/PPiRS 
to the individual contracting professional. Contracting officers and source 
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selection teams are allowed by the Federal Acquisition Regulation (FAR) to use 
some subjectivity in determining how much to consider contractor past 
performance. Knowing how much they value the product will aid us in 
determining whether improvements to the system might be beneficial and 
whether the quality of the system affects how much value is assigned to it. 

Which provides more impact, the objective scores or the narratives? 

The purpose of this question is to compare the two sections of the CPARS 
Report Card in terms of value to the contracting professional. This question is 
carefully worded to that it is a zero sum value pool split between the two 
sections. We are not asking how much total value each section provides 
independent of each other. 

What ideas do you think wouid increase the impact of contractor 
past performance information in source seiection and/or contract 
administration? 

The purpose of this question is to get opinions from the interview subject. 
Since evaluating past performance information is mandated by the FAR, our 
research is attempting to find ways in which CPARS/PPiRS can be made more 
user-friendly and responsive for this purpose. Opinions from experts in the field 
will be very useful. 

Do you feei that past performance information is weighted 
appropriateiy in source seiection? 

The purpose of this question is to determine whether the interview subject 
believes that the FAR or their agency puts too much or too little emphasis on 
contractor past performance. If the interview subject believes there is too much 
emphasis on past performance, it might be because the CPARS/PPIRS systems 
are insufficient for providing useful information. This question was worded to 
avoid the injection of bias. The data obtained from the answers to this question 
will be particularly useful when combined with the information from the other 
answers received. 
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Do you find that the narrative usuaiiy provides usefui information 
above and beyond the objective scores? 

The purpose of this question is to determine whether the CPARS narrative 
section has vaiue whether or not the objective scores have more value than the 
narrative. It is possible that the interview subject will answer yes even if he or 
she answered to a previous question that the objective scores are more impactful 
than the narrative. 

If the objective scores of past performance information were to be 
eiiminated, how wouid that impact the source seiection or contract 
administration process? 

The purpose of this question is to establish the absolute value of the 
CPARS objective scores. It allows the interview subject the opportunity to give a 
bottom line answer to the question: How much does this really matter? The 
answers to this question are going to be particularly useful when compared to 
answers to previous interview questions. If the interviewee finds that the 
narrative section is much more impactful than the objective scores then the 
conclusion will be far different than if the opposite is true. 

If the narrative portion of past performance information were to be 
eiiminated, how wouid that impact the source seiection or contract 
administration process? 

The purpose of this question is the same as the previous question except 
for the narrative portion of the CPARS Report Card. 
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